<div dir="ltr">It's not <font face="arial, sans-serif">synchronous, each request/eventlet co-rotine will yield/trampoline back to the reactor/hub on every socket operation that raises EWOULDBLOCK. In cases where there's a tight long running read/write loop you'll normally find a call to eventlet.sleep (or in at least one case a queue) to avoid starvation.</font><div>
<font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Tuning workers and concurrency has a lot to do with the hardware and some with the work-load. The processing rate of an individual proxy server is mostly cpu bound and depends on if you're doing ssl termination in front of your proxy. Request rate throughput is easily scaled by adding more proxy servers (assuming your client doesn't bottleneck, look to <a href="https://github.com/swiftstack/ssbench">https://github.com/swiftstack/ssbench</a> for a decently scalable Swift benchmark suite). Throughput is harder to scale wide because of load balancing - round-robin dns seems to be a good choice, or ssbench has an option to benchmark a set of storage urls (list of proxy servers).</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Have you read:</font></div><div><font face="arial, sans-serif"><a href="http://docs.openstack.org/developer/swift/deployment_guide.html#general-service-tuning">http://docs.openstack.org/developer/swift/deployment_guide.html#general-service-tuning</a></font></div>
<div><div><span style="font-family:arial,sans-serif;font-size:12.800000190734863px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:12.800000190734863px">-Clay</span></div></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Mon, Jan 13, 2014 at 12:21 AM, Kuo Hugo <span dir="ltr"><<a href="mailto:tonytkdk@gmail.com" target="_blank">tonytkdk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Shrinand, <div><br></div><div>The concurrency bottleneck of Swift cluster could be various. </div><div>Here's a list : </div><div><ul><li>Settings of each workers, workers count, max_clients, threads_per_disk. <br>
</li><li>Proxy CPU bound<br></li><li>Storage nodes CPU bound<br></li><li>Total Disk IO capacity (includes available memory for xfs caching)<br></li><li>The power of your client machines</li><li>Network issue</li></ul><div>
<br></div></div><div>You need to analyze the monitoring data to find the real bottleneck. </div><div>The range of concurrency connections performance depends on the deployment. </div><div>Concurrent connections from 150(VMs) to 6K+(physical sever pool). Of course that you can setup multiple proxy servers for handling higher concurrency as long as your storage nodes can stand for it. </div>
<div><br></div><div>The path of a request in my knowing: </div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>Client --> Proxy-server --> object-server --> container-server (optional async) --> object-server --> Proxy-server --> Client --> close connection.</div>
</blockquote><div><br></div><div>Hope it help</div><div>Hugo </div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
2014/1/11 Shrinand Javadekar <span dir="ltr"><<a href="mailto:shrinand@maginatics.com" target="_blank">shrinand@maginatics.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div class="h5">
<div dir="ltr">Hi,<div><br></div><div>This question is specific to Openstack Swift. I am trying to understand just how much is the proxy server a bottleneck when multiple clients are concurrently trying to write to a swift cluster. Has anyone done experiments to measure this? It'll be great to see some results.</div>
<div><br></div><div>I see that the proxy-server already has a "workers" config option. However, looks like that is the # of threads in one proxy-server process. Does having multiple proxy-servers themselves, running on different nodes (and having some load-balancer in front of them) help in satisfying more concurrent writes? Or will these multiple proxy-servers also get bottlenecked on the account/container/obj server?</div>
<div><br></div><div>Also, looking at the code in swift/proxy-server/controllers/obj.py, it seems that each request that the proxy-server sends to the backend servers (account/container/obj) is synchronous. It does not send the request and go back do accept more requests. Is this one of the reasons why write requests can be slow?</div>
<div><br></div><div>Thanks in advance.</div><div>-Shri</div></div>
<br></div></div>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div><br></div>