<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 25 February 2015 at 02:35, Robert Collins <span dir="ltr"><<a href="mailto:robertc@robertcollins.net" target="_blank">robertc@robertcollins.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 24 February 2015 at 01:07, Salvatore Orlando <<a href="mailto:sorlando@nicira.com">sorlando@nicira.com</a>> wrote:<br>
> Lazy-Stacker summary:<br>
...<br>
<span class="">> In the medium term, there are a few things we might consider for Neutron's<br>
> "built-in IPAM".<br>
> 1) Move the allocation logic out of the driver, thus making IPAM an<br>
> independent service. The API workers will then communicate with the IPAM<br>
> service through a message bus, where IP allocation requests will be<br>
> "naturally serialized"<br>
> 2) Use 3-party software as dogpile, zookeeper but even memcached to<br>
> implement distributed coordination. I have nothing against it, and I reckon<br>
> Neutron can only benefit for it (in case you're considering of arguing that<br>
> "it does not scale", please also provide solid arguments to support your<br>
> claim!). Nevertheless, I do believe API request processing should proceed<br>
> undisturbed as much as possible. If processing an API requests requires<br>
> distributed coordination among several components then it probably means<br>
> that an asynchronous paradigm is more suitable for that API request.<br>
<br>
</span>So data is great. It sounds like as long as we have an appropriate<br>
retry decorator in place, that write locks are better here, at least<br>
for up to 30 threads. But can we trust the data?<br></blockquote><div><br></div><div>Not unless you can prove the process to obtain them is correct.</div><div>Otherwise we'd still think the sun rotates around the earth.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
One thing I'm not clear on is the SQL statement count. You say 100<br>
queries for A-1 with a time on Galera of 0.06*1.2=0.072 seconds per<br>
allocation ? So is that 2 queries over 50 allocations over 20 threads?<br></blockquote><div><br></div><div>So the query number reported in the thread is for a single node test. The numbers for the galera tests are on github, and if you have a galera environment you can try and run the experiment there too.</div><div>The algorithm indeed should perform a single select query for each IP allocation and the number appears to be really too high. It is coming from sqlalchemy hooks, so I guess it's reliable. It's worth noting that I put the count for all queries, including those for setting up the environment, and verifying the algorithm successful completion, so those should be removed. I can easily enable debug logging and provide a detailed breakdown of db operations for every algorithm.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I'm not clear on what the request parameter in the test json files<br>
does, and AFAICT your threads each do one request each. As such I<br>
suspect that you may be seeing less concurrency - and thus contention<br>
- than real-world setups where APIs are deployed to run worker<br>
processes in separate processes and requests are coming in<br>
willy-nilly. The size of each algorithms workload is so small that its<br>
feasible to imagine the thread completing before the GIL bytecount<br>
code trigger (see<br>
<a href="https://docs.python.org/2/library/sys.html#sys.setcheckinterval" target="_blank">https://docs.python.org/2/library/sys.html#sys.setcheckinterval</a>) and<br>
the GIL's lack of fairness would exacerbate that.<br></blockquote><div><br></div><div>I have a retry counter which testifies that contention is actually occurring.</div><div>Indeed algorithms which do sequential allocation see a lot of contention, so I do not think that I'm just fooling myself and the tests are actually running serially!</div><div>Anyway, the multiprocess suggestion is very valid and I will repeat the experiments (I'm afraid that won't happen before Friday), because I did not consider the GIL aspect you mention, as I dumbly expected that python will simple spawn a different pthread for each thread and let the OS do the scheduling.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
If I may suggest:<br>
- use multiprocessing or some other worker-pool approach rather than threads<br>
- or set setcheckinterval down low (e.g. to 20 or something)<br>
- do multiple units of work (in separate transactions) within each<br>
worker, aim for e.g. 10 seconds or work or some such.<br></blockquote><div><br></div><div>This last suggestion also makes sense.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- log with enough detail that we can report on the actual concurrency<br>
achieved. E.g. log the time in us when each transaction starts and<br>
finishes, then we can assess how many concurrent requests were<br>
actually running.<br></blockquote><div><br></div><div>I put simple output on github only, but full debug logging can be achieved by simply changing a constant.</div><div>However, I'm collecting the number of retries for each thread as an indirect marker of concurrency level. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
If the results are still the same - great, full steam ahead. If not,<br>
well lets revisit :)<br></blockquote><div><br></div><div>Obviously. We're not religious here. We'll simply do what the data suggest as the best way forward. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
-Rob<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
--<br>
Robert Collins <<a href="mailto:rbtcollins@hp.com">rbtcollins@hp.com</a>><br>
Distinguished Technologist<br>
HP Converged Cloud<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</font></span></blockquote></div><br></div></div>