<div dir="ltr">Hi folks,<div><br></div><div>There was a small discussion about the better way of doing sql operations for vni synchronization with the config.</div><div>Initial proposal was to handle those in chunks. Carl also suggested to issue a single sql query.</div>
<div>I've did some testing with my sql and postgress.</div><div>I've tested the following scenario: vxlan range is changed from 50000:150000 to 0:100000 and vice versa.</div><div>That involves adding and deleting 50000 vni in each test.</div>
<div><br></div><div>Here are the numbers:</div><div><table cellspacing="0" cellpadding="0" dir="ltr" border="1" style="table-layout:fixed;font-size:13px;font-family:arial,sans,sans-serif;border-collapse:collapse;border:1px solid rgb(204,204,204)">
<colgroup><col width="137"><col width="122"><col width="117"><col width="82"><col width="128"><col width="133"><col width="100"></colgroup><tbody><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom;font-weight:bold;text-align:center">
50k vnis to add/delete</td><td style="padding:2px 3px;vertical-align:bottom;font-weight:bold;text-align:center">Pg adding vnis</td><td style="padding:2px 3px;vertical-align:bottom;font-weight:bold;text-align:center">Pg deleting vnis</td>
<td style="padding:2px 3px;vertical-align:bottom;font-weight:bold;text-align:center">Pg Total</td><td style="padding:2px 3px;vertical-align:bottom;font-weight:bold;text-align:center">Mysql adding vnis</td><td style="padding:2px 3px;vertical-align:bottom;font-weight:bold;text-align:center">
Mysql deleting vnis</td><td style="padding:2px 3px;vertical-align:bottom;font-weight:bold;text-align:center">Mysql total</td></tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom">non-chunked sql</td>
<td style="padding:2px 3px;vertical-align:bottom;text-align:right">23</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">22</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">45</td>
<td style="padding:2px 3px;vertical-align:bottom;text-align:right">14</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">20</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">34</td>
</tr><tr style="height:21px"><td style="padding:2px 3px;vertical-align:bottom">chunked in 100</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">20</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">
17</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">37</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">14</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">
17</td><td style="padding:2px 3px;vertical-align:bottom;text-align:right">31</td></tr></tbody></table></div><div><br></div><div>I've done about 5 tries to get each number to minimize random floating factor (due to swaps, disc or cpu activity or other factors)</div>
<div>That might be surprising that issuing multiple sql statements instead one big is little bit more efficient, so I would appreciate if someone could reproduce those numbers.</div><div>Also I'd like to note that part of code that iterates over vnis fetched from db is taking 10 seconds both on mysql and postgress and is a part of "deleting vnis" numbers.</div>
<div>In other words, difference between multiple DELETE sql statements and single one is even bigger (in percent) than these numbers show.</div><div><br></div><div>The code which I used to test is here: <a href="http://paste.openstack.org/show/83298/">http://paste.openstack.org/show/83298/</a></div>
<div>Right now the chunked version is commented out, so to switch between versions some lines should be commented and some - uncommented.</div><div><br></div><div>Thanks,</div><div>Eugene.</div><div><br></div><div><br></div>
<div>P.S. I'm also afraid that issuing one big sql statement (and it will be megabytes big) could lead to timeouts/deadlocks just because it will take too much time, how ever I'm not 100% sure about that, it's just my bare concern.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 5, 2014 at 1:06 PM, Xurong Yang <span dir="ltr"><<a href="mailto:idopra@gmail.com" target="_blank">idopra@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">great.<div>I will do more test base on <span style="font-family:arial,sans-serif;font-size:14px;white-space:nowrap">Eugene Nikanorov's modification.</span></div><div><font face="arial, sans-serif"><span style="font-size:14px;white-space:nowrap"><b><br>
</b></span></font></div><div><font face="arial, sans-serif"><span style="font-size:14px;white-space:nowrap"><b>Thanks,</b></span></font></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-06-05 11:01 GMT+08:00 Isaku Yamahata <span dir="ltr"><<a href="mailto:isaku.yamahata@gmail.com" target="_blank">isaku.yamahata@gmail.com</a>></span>:<div>
<div class="h5"><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Wow great.<br>
I think the same applies to gre type driver.<br>
so we should create similar one after vxlan case is resolved.<br>
<br>
thanks,<br>
<div><div><br>
<br>
On Thu, Jun 05, 2014 at 12:36:54AM +0400,<br>
Eugene Nikanorov <<a href="mailto:enikanorov@mirantis.com" target="_blank">enikanorov@mirantis.com</a>> wrote:<br>
<br>
> We hijacked the vxlan initialization performance thread with ipam! :)<br>
> I've tried to address initial problem with some simple sqla stuff:<br>
> <a href="https://review.openstack.org/97774" target="_blank">https://review.openstack.org/97774</a><br>
> With sqlite it gives ~3x benefit over existing code in master.<br>
> Need to do a little bit more testing with real backends to make sure<br>
> parameters are optimal.<br>
><br>
> Thanks,<br>
> Eugene.<br>
><br>
><br>
> On Thu, Jun 5, 2014 at 12:29 AM, Carl Baldwin <<a href="mailto:carl@ecbaldwin.net" target="_blank">carl@ecbaldwin.net</a>> wrote:<br>
><br>
> > Yes, memcached is a candidate that looks promising. First things first,<br>
> > though. I think we need the abstraction of an ipam interface merged. That<br>
> > will take some more discussion and work on its own.<br>
> ><br>
> > Carl<br>
> > On May 30, 2014 4:37 PM, "Eugene Nikanorov" <<a href="mailto:enikanorov@mirantis.com" target="_blank">enikanorov@mirantis.com</a>><br>
> > wrote:<br>
> ><br>
> >> > I was thinking it would be a separate process that would communicate over<br>
> >> the RPC channel or something.<br>
> >> memcached?<br>
> >><br>
> >> Eugene.<br>
> >><br>
> >><br>
> >> On Sat, May 31, 2014 at 2:27 AM, Carl Baldwin <<a href="mailto:carl@ecbaldwin.net" target="_blank">carl@ecbaldwin.net</a>> wrote:<br>
> >><br>
> >>> Eugene,<br>
> >>><br>
> >>> That was part of the "whole new set of complications" that I<br>
> >>> dismissively waved my hands at. :)<br>
> >>><br>
> >>> I was thinking it would be a separate process that would communicate<br>
> >>> over the RPC channel or something. More complications come when you<br>
> >>> think about making this process HA, etc. It would mean going over RPC<br>
> >>> to rabbit to get an allocation which would be slow. But the current<br>
> >>> implementation is slow. At least going over RPC is greenthread<br>
> >>> friendly where going to the database doesn't seem to be.<br>
> >>><br>
> >>> Carl<br>
> >>><br>
> >>> On Fri, May 30, 2014 at 2:56 PM, Eugene Nikanorov<br>
> >>> <<a href="mailto:enikanorov@mirantis.com" target="_blank">enikanorov@mirantis.com</a>> wrote:<br>
> >>> > Hi Carl,<br>
> >>> ><br>
> >>> > The idea of in-memory storage was discussed for similar problem, but<br>
> >>> might<br>
> >>> > not work for multiple server deployment.<br>
> >>> > Some hybrid approach though may be used, I think.<br>
> >>> ><br>
> >>> > Thanks,<br>
> >>> > Eugene.<br>
> >>> ><br>
> >>> ><br>
> >>> > On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin <<a href="mailto:carl@ecbaldwin.net" target="_blank">carl@ecbaldwin.net</a>><br>
> >>> wrote:<br>
> >>> >><br>
> >>> >> This is very similar to IPAM... There is a space of possible ids or<br>
> >>> >> addresses that can grow very large. We need to track the allocation<br>
> >>> >> of individual ids or addresses from that space and be able to quickly<br>
> >>> >> come up with a new allocations and recycle old ones. I've had this in<br>
> >>> >> the back of my mind for a week or two now.<br>
> >>> >><br>
> >>> >> A similar problem came up when the database would get populated with<br>
> >>> >> the entire free space worth of ip addresses to reflect the<br>
> >>> >> availability of all of the individual addresses. With a large space<br>
> >>> >> (like an ip4 /8 or practically any ip6 subnet) this would take a very<br>
> >>> >> long time or never finish.<br>
> >>> >><br>
> >>> >> Neutron was a little smarter about this. It compressed availability<br>
> >>> >> in to availability ranges in a separate table. This solved the<br>
> >>> >> original problem but is not problem free. It turns out that writing<br>
> >>> >> database operations to manipulate both the allocations table and the<br>
> >>> >> availability table atomically is very difficult and ends up being very<br>
> >>> >> slow and has caused us some grief. The free space also gets<br>
> >>> >> fragmented which degrades performance. This is what led me --<br>
> >>> >> somewhat reluctantly -- to change how IPs get recycled back in to the<br>
> >>> >> free pool which hasn't been very popular.<br>
> >>> >><br>
> >>> >> I wonder if we can discuss a good pattern for handling allocations<br>
> >>> >> where the free space can grow very large. We could use the pattern<br>
> >>> >> for the allocation of both IP addresses, VXlan ids, and other similar<br>
> >>> >> resource spaces.<br>
> >>> >><br>
> >>> >> For IPAM, I have been entertaining the idea of creating an allocation<br>
> >>> >> agent that would manage the availability of IPs in memory rather than<br>
> >>> >> in the database. I hesitate, because that brings up a whole new set<br>
> >>> >> of complications. I'm sure there are other potential solutions that I<br>
> >>> >> haven't yet considered.<br>
> >>> >><br>
> >>> >> The L3 subteam is currently working on a pluggable IPAM model. Once<br>
> >>> >> the initial framework for this is done, we can more easily play around<br>
> >>> >> with changing the underlying IPAM implementation.<br>
> >>> >><br>
> >>> >> Thoughts?<br>
> >>> >><br>
> >>> >> Carl<br>
> >>> >><br>
> >>> >> On Thu, May 29, 2014 at 4:01 AM, Xurong Yang <<a href="mailto:idopra@gmail.com" target="_blank">idopra@gmail.com</a>><br>
> >>> wrote:<br>
> >>> >> > Hi, Folks,<br>
> >>> >> ><br>
> >>> >> > When we configure VXLAN range [1,16M], neutron-server service costs<br>
> >>> long<br>
> >>> >> > time and cpu rate is very high(100%) when initiation. One test base<br>
> >>> on<br>
> >>> >> > postgresql has been verified: more than 1h when VXLAN range is [1,<br>
> >>> 1M].<br>
> >>> >> ><br>
> >>> >> > So, any good solution about this performance issue?<br>
> >>> >> ><br>
> >>> >> > Thanks,<br>
> >>> >> > Xurong Yang<br>
> >>> >> ><br>
> >>> >> ><br>
> >>> >> ><br>
> >>> >> > _______________________________________________<br>
> >>> >> > OpenStack-dev mailing list<br>
> >>> >> > <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> >>> >> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> >>> >> ><br>
> >>> >><br>
> >>> >> _______________________________________________<br>
> >>> >> OpenStack-dev mailing list<br>
> >>> >> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> >>> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> >>> ><br>
> >>> ><br>
> >>> ><br>
> >>> > _______________________________________________<br>
> >>> > OpenStack-dev mailing list<br>
> >>> > <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> >>> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> >>> ><br>
> >>><br>
> >>> _______________________________________________<br>
> >>> OpenStack-dev mailing list<br>
> >>> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> >>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> >>><br>
> >><br>
> >><br>
> >> _______________________________________________<br>
> >> OpenStack-dev mailing list<br>
> >> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> >><br>
> >><br>
> > _______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> ><br>
<br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
</div></div><span><font color="#888888">--<br>
Isaku Yamahata <<a href="mailto:isaku.yamahata@gmail.com" target="_blank">isaku.yamahata@gmail.com</a>><br>
</font></span><div><div><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div></div></div><br></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>