<div dir="ltr">Can you start a bug on launchpad and upload the conntrack attachment to the bug?<div><br></div><div>Switching to the rootwrap daemon should also help significantly.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 11, 2017 at 12:32 PM, Ajay Kalambur (akalambu) <span dir="ltr"><<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">



<div style="word-wrap:break-word">
<div>
<div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
Hi Kevin</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
The information you asked for</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
For 1 compute node with 45 Vms here is the number of connection tracking entries getting deleted</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div>cat conntrack.file  | wc -l</div>
<div>   38528</div>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
The file with output is 14MB so ill email it to Ian and he can share it if needed</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
Security group rules</div>
<div>
<div><font face="Calibri,sans-serif"><span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"></span>Direction<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>Ether Type<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span>IP Protocol<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>Port Range<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span>Remote IP Prefix<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>Remote Security Group<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>Actions</font></div>
<div><font face="Calibri,sans-serif">Egress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>IPv4<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span><a href="http://0.0.0.0/0" target="_blank">0.0.0.0/0</a><span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span></font></div>
<div><font face="Calibri,sans-serif">Ingress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>IPv6<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span>-<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>default<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span></font></div>
<div><font face="Calibri,sans-serif">Egress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>IPv6<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span>::/0<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>-<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span></font></div>
<div><font face="Calibri,sans-serif">Ingress<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>IPv4<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span>Any<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap"> </span>-<span class="m_4511050849682920103Apple-tab-span" style="white-space:pre-wrap">
</span></font></div>
</div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">Please let me know if u need the dump of conntrack entries if so I can email it to email address of your choice</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">Ajay</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div id="m_4511050849682920103MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<br>
</div>
<span id="m_4511050849682920103OLK_SRC_BODY_SECTION" style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div style="font-family:Calibri;font-size:12pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Ajay Kalambur <<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.<wbr>openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Monday, September 11, 2017 at 10:02 AM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.<wbr>openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [neutron]OVS connection tracking cleanup<br>
</div><div><div class="h5">
<div><br>
</div>
<span>
<div>
<div dir="auto">
<div>Hi Kevin</div>
<div id="m_4511050849682920103AppleMailSignature">Thanks for your response it was about 50 vms </div>
<div id="m_4511050849682920103AppleMailSignature">Ajay<br>
<br>
<br>
</div>
<div><br>
On Sep 11, 2017, at 9:49 AM, Kevin Benton <<a href="mailto:kevin@benton.pub" target="_blank">kevin@benton.pub</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">The biggest improvement will be switching to native netlink calls: <a href="https://review.openstack.org/#/c/470912/" target="_blank">https://review.<wbr>openstack.org/#/c/470912/</a>
<div><br>
</div>
<div>How many VMs were on a single compute node?</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Sep 11, 2017 at 9:15 AM, Ajay Kalambur (akalambu)
<span dir="ltr"><<a href="mailto:akalambu@cisco.com" target="_blank">akalambu@cisco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">Hi</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">I am performing a scale test and I see that after creating 500 Vms with ping traffic between them it took almost 1 hr for the connection tracking </div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">To clean up and ovs agent was busy doing this and unable to service any new port bind requests on some computes for almost an hr </div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">It took that long for conntrack clean up to complete</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">I see the following bug</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><a href="https://bugs.launchpad.net/neutron/+bug/1513765" target="_blank">https://bugs.launchpad.net/neu<wbr>tron/+bug/1513765</a></div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br>
</div>
<div>And I also have the fix below</div>
<div><a href="https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7aeb8dd4b1d122e17eef8687192cd122b79fd6e" target="_blank">https://git.openstack.org/cgit<wbr>/openstack/neutron/commit/?id=<wbr>d7aeb8dd4b1d122e17eef8687192cd<wbr>122b79fd6e</a></div>
<div><br>
</div>
<div><br>
</div>
<div>Still see really long times for conntrack cleanup</div>
<div><br>
</div>
<div>What is the solution to this problem in scale scenarios?</div>
<span class="m_4511050849682920103HOEnZb"><font color="#888888">
<div>Ajay</div>
<div><br>
</div>
<div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">
<div id="m_4511050849682920103m_-6662988751102660720MAC_OUTLOOK_SIGNATURE"></div>
</div>
</font></span></div>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>______________________________<wbr>______________________________<wbr>______________</span><br>
<span>OpenStack Development Mailing List (not for usage questions)</span><br>
<span>Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org</a>?subject:<wbr>unsubscribe</span><br>
<span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a></span><br>
</div>
</blockquote>
</div>
</div>
</span></div></div></span>
</div>

<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div>