<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>It looks like the conntrack deletion can be skipped for port deletion no?</div>
<div>On bulk deletes of lot of Vms the entries that were deleted never existed in conntrack table</div>
</div>
</div>
<div><br>
</div>
<div>From looking the patch below seems to go along those lines</div>
<div><a href="https://review.openstack.org/#/c/243994">https://review.openstack.org/#/c/243994</a>/</div>
<div><br>
</div>
<div>Is there a plan to distinguish between port deletes and port updates when it comes to conntrack rule deletions because in a scale scenario on OVS VLAN this is really a blocker for back to back scale tests being run</div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Ajay Kalambur <<a href="mailto:akalambu@cisco.com">akalambu@cisco.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, September 27, 2017 at 4:42 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Cc: </span>"Ian Wells (iawells)" <<a href="mailto:iawells@cisco.com">iawells@cisco.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [neutron]OVS connection tracking cleanup<br>
</div>
<div><br>
</div>
<span style="mso-bookmark:_MailOriginalBody">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>Also the weird part with this conntrack deletion I perform a conntrack –L to view the table I see no entry for any of the entries its trying to delete. Those entries are all removed anyways when Vms are cleaned up from the look of it. So it looks like
all those conntrack deletions were pretty much no-ops</div>
<div>Ajay</div>
<div><br>
</div>
<div>
<div id=""></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Ajay Kalambur <<a href="mailto:akalambu@cisco.com">akalambu@cisco.com</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, September 12, 2017 at 9:30 AM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Cc: </span>"Ian Wells (iawells)" <<a href="mailto:iawells@cisco.com">iawells@cisco.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [neutron]OVS connection tracking cleanup<br>
</div>
<div><br>
</div>
<span style="mso-bookmark:_MailOriginalBody">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>Hi Kevin</div>
<div>Sure will log a bug</div>
<div>Also does the config change involve having both these lines in the neutron.conf file?</div>
<div>
<div>[agent]</div>
<div>root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf</div>
<div>root_helper_daemon = sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf</div>
</div>
<div><br>
</div>
<div>If I have only the second line I see the exception below on neutron openvswitch agent bring up:</div>
<div><br>
</div>
<div>
<div>2017-09-12 09:23:03.633 35 DEBUG neutron.agent.linux.utils [req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] Running command: ['ps', '--ppid', '103', '-o', 'pid='] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:89</div>
<div>2017-09-12 09:23:03.762 35 ERROR ryu.lib.hub [req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] hub: uncaught exception: Traceback (most recent call last):</div>
<div> File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch</div>
<div> return func(*args, **kwargs)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 42, in agent_main_wrapper</div>
<div> ovs_agent.main(bridge_classes)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2184, in main</div>
<div> agent.daemon_loop()</div>
<div> File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper</div>
<div> return f(*args, **kwargs)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2100, in daemon_loop</div>
<div> self.ovsdb_monitor_respawn_interval) as pm:</div>
<div> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__</div>
<div> return self.gen.next()</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 35, in get_polling_manager</div>
<div> pm.start()</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 57, in start</div>
</div>
<div>
<div> while not self.is_active():</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 100, in is_active</div>
<div> self.pid, self.cmd_without_namespace)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 159, in pid</div>
<div> run_as_root=self.run_as_root)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 297, in get_root_helper_child_pid</div>
<div> pid = find_child_pids(pid)[0]</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 179, in find_child_pids</div>
<div> log_fail_as_error=False)</div>
<div> File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 128, in execute</div>
<div> _stdout, _stderr = obj.communicate(_process_input)</div>
<div> File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate</div>
<div> return self._communicate(input)</div>
<div> File "/usr/lib64/python2.7/subprocess.py", line 1403, in _communicate</div>
<div> stdout, stderr = self._communicate_with_select(input)</div>
<div> File "/usr/lib64/python2.7/subprocess.py", line 1504, in _communicate_with_select</div>
<div> rlist, wlist, xlist = select.select(read_set, write_set, [])</div>
<div> File "/usr/lib/python2.7/site-packages/eventlet/green/select.py", line 86, in select</div>
<div> return hub.switch()</div>
<div> File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch</div>
<div> return self.greenlet.switch()</div>
<div>Timeout: 5 seconds</div>
<div><br>
</div>
<div>2017-09-12 09:23:03.860 35 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=95</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Ajay</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div id=""></div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Kevin Benton <<a href="mailto:kevin@benton.pub">kevin@benton.pub</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Monday, September 11, 2017 at 1:12 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Cc: </span>"Ian Wells (iawells)" <<a href="mailto:iawells@cisco.com">iawells@cisco.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [neutron]OVS connection tracking cleanup<br>
</div>
<div><br>
</div>
<span style="mso-bookmark:_MailOriginalBody">
<div>
<div>
<div dir="ltr">Can you start a bug on launchpad and upload the conntrack attachment to the bug?
<div><br>
</div>
<div>Switching to the rootwrap daemon should also help significantly.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Sep 11, 2017 at 12:32 PM, Ajay Kalambur (akalambu)
<span dir="ltr"><<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>
<div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">Hi Kevin</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">The information you asked for</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">For 1 compute node with 45 Vms here is the number of connection tracking entries getting deleted</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div>cat conntrack.file | wc -l</div>
<div> 38528</div>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">The file with output is 14MB so ill email it to Ian and he can share it if needed</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">Security group rules</div>
<div>
<div><font face="Calibri,sans-serif"><span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Direction<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Ether Type<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>IP
Protocol<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Port Range<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Remote IP Prefix<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Remote
Security Group<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Actions</font></div>
<div><font face="Calibri,sans-serif">Egress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>IPv4<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span><a href="http://0.0.0.0/0" target="_blank">0.0.0.0/0</a><span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span></font></div>
<div><font face="Calibri,sans-serif">Ingress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>IPv6<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>-<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>default<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span></font></div>
<div><font face="Calibri,sans-serif">Egress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>IPv6<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>::/0<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>-<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span></font></div>
<div><font face="Calibri,sans-serif">Ingress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>IPv4<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>-<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span></font></div>
</div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">Please let me know if u need the dump of conntrack entries if so I can email it to email address of your choice</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">Ajay</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div id="m_4511050849682920103MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<span id="m_4511050849682920103OLK_SRC_BODY_SECTION" style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div style="font-family:Calibri;font-size:12pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Ajay Kalambur <<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.<wbr>openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Monday, September 11, 2017 at 10:02 AM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.<wbr>openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [neutron]OVS connection tracking cleanup<br>
</div>
<div>
<div class="h5">
<div><br>
</div>
<span>
<div>
<div dir="auto">
<div>Hi Kevin</div>
<div id="m_4511050849682920103AppleMailSignature">Thanks for your response it was about 50 vms </div>
<div id="m_4511050849682920103AppleMailSignature">Ajay<br>
<br>
<br>
</div>
<div><br>
On Sep 11, 2017, at 9:49 AM, Kevin Benton <<a href="mailto:kevin@benton.pub" target="_blank">kevin@benton.pub</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">The biggest improvement will be switching to native netlink calls: <a href="https://review.openstack.org/#/c/470912/" target="_blank">https://review.<wbr>openstack.org/#/c/470912/</a>
<div><br>
</div>
<div>How many VMs were on a single compute node?</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Sep 11, 2017 at 9:15 AM, Ajay Kalambur (akalambu)
<span dir="ltr"><<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">Hi</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">I am performing a scale test and I see that after creating 500 Vms with ping traffic between them it took almost 1 hr for the connection tracking </div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">To clean up and ovs agent was busy doing this and unable to service any new port bind requests on some computes for almost an hr </div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">It took that long for conntrack clean up to complete</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">I see the following bug</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><a href="https://bugs.launchpad.net/neutron/+bug/1513765" target="_blank">https://bugs.launchpad.net/neu<wbr>tron/+bug/1513765</a></div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div>And I also have the fix below</div>
<div><a href="https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7aeb8dd4b1d122e17eef8687192cd122b79fd6e" target="_blank">https://git.openstack.org/cgit<wbr>/openstack/neutron/commit/?id=<wbr>d7aeb8dd4b1d122e17eef8687192cd<wbr>122b79fd6e</a></div>
<div><br>
</div>
<div><br>
</div>
<div>Still see really long times for conntrack cleanup</div>
<div><br>
</div>
<div>What is the solution to this problem in scale scenarios?</div>
<span class="m_4511050849682920103HOEnZb"><font color="#888888">
<div>Ajay</div>
<div><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div id="m_4511050849682920103m_-6662988751102660720MAC_OUTLOOK_SIGNATURE"></div>
</div>
</font></span></div>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>______________________________<wbr>______________________________<wbr>______________</span><br>
<span>OpenStack Development Mailing List (not for usage questions)</span><br>
<span>Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">
OpenStack-dev-request@lists.<wbr>openstack.org</a>?subject:<wbr>unsubscribe</span><br>
<span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a></span><br>
</div>
</blockquote>
</div>
</div>
</span></div>
</div>
</span></div>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span></span></div>
</div>
</span></span></div>
</div>
</span></span>
</body>
</html>