<div dir="ltr">Thanks for the inputs Chuck. Please see my responses inline.
<div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Oct 10, 2013 at 7:56 AM, Chuck Thier <span dir="ltr"><<a href="mailto:cthier@gmail.com" target="_blank">cthier@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Shri,<div><br></div><div>I think your observations are fairly spot on. Here are a couple of thoughts/comments.</div><div><br></div><div>1. I wonder if you are maxing out how much your client can push at 128 threads. If you were to increase the number threads (or number of clients) for the higher container counts, you could get more transactions through.</div>
</div></blockquote><div> </div><div>[SJ] In this experiment I simply wanted to see how sharding across containers helps gives the same input data rate. I will run some numbers with higher number of threads to see what's the max number of operations per second I can get.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">
<div><br></div><div>2. Cloudfiles rate limits PUTs at 100 per second to a single container. This helps ensure fairly consistent performance to a single container. We also put our container data on SSD drives to help drive better performance. So your max theoretical performance is 100xNUM_CONTAINERS PUTs/sec.</div>
</div></blockquote><div><br></div><div>[SJ] Great to know about the SSDs and the rate limits. Is it also possible to know what version of Swift has been deployed at Rackspace Cloudfiles?</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div><br></div><div>3. It would be worthwhile to test even larger containers to test how much container size affects performance. I don't think your sample size is large enough.</div></div></blockquote><div><br></div>
<div>[SJ] Yeah, especially with SSDs, this is definitely not a large enough sample. I guess, I should start at 1M and go upto 10M or so.</div><div> </div><div>Will keep you'll posted.</div><div><br></div><div>-Shri</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><div class="gmail_quote">On Wed, Oct 9, 2013 at 11:11 PM, Shrinand Javadekar <span dir="ltr"><<a href="mailto:shrinand@maginatics.com" target="_blank">shrinand@maginatics.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Thanks Chuck.</div><div><br></div>In order to really measure this, I ran some tests on Rackspace; i.e. I got a VM on Rackspace and that VM was talking to a Rackspace Cloudfiles-US swift cluster. The VM and object store were both in the Chicago region. The downside of using a public object store is that I have little idea about the configuration of Swift being used. But installing and configuring one's own enterprise class Swift cluster is no child's play either (to put it mildly :D).<div>
<br></div><div>In the first experiment, 128 threads were continuously trying to write 1 byte blobs into N containers where N was in (1, 32, 64, 128, 256, 512). The experiment ran for 15 minutes. The experiment was run thrice for each N and the results below are the average of three runs.</div>
<div><br></div><div><img src="http://on.magfs.io/19mtr49" alt="Inline image 1"><br></div><div>The number of writes completed in 15 minutes if ~87K for a single container, whereas when these writes are sharded across 32 containers, this # is ~135K.</div>
<div><br></div><div>The second experiment was to find out whether Swift becomes slower as the number of objects in a container increases. To do this, I measured the time it was taking to write blobs in a single container. Here again, I ran the experiment three times and the graph below is the average of the three runs.</div>
<div><br></div><div><img src="http://on.magfs.io/19jfORm" alt="Inline image 2"><br></div><div><br></div><div>If a container has less than 1.6M blobs, the average time to write a blob is ~12.58ms whereas if the container has > 1.6M blobs, the average time to write a blob is ~13.29ms. The trend definitely seems to be that as number of objects increase, the time to write also increases.</div>
<div><br></div><div>I guess the absolute number may differ depending on factors like memory, CPU, disk (SSD's vs rotational) of the servers running swift. But the relative numbers give a better picture of the benefits of: </div>
<div><br></div><div>i) Sharding across containers to increase throughput</div><div>ii) Restricting the number of objects per container</div><div><br></div><div>Let me know if I have missed out on anything or if there are more experiments to run that would make Swift #awesome!!</div>
<div><br></div><div>-Shri</div><div><br></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 3, 2013 at 7:47 AM, Chuck Thier <span dir="ltr"><<a href="mailto:cthier@gmail.com" target="_blank">cthier@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Shri,<div><br></div><div>The short answer is that sharding your data across containers in swift is generally a good idea.</div>
<div><br></div><div>The limitations with containers has a lot more to do with overall concurrency rather than total objects in a container. The number of objects in a container can have an affect on that, but will be less of an issue if you are not putting objects in at a high concurrency.</div>
<div><br></div><div>--</div><div>Chuck</div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Sun, Sep 1, 2013 at 9:39 PM, Shrinand Javadekar <span dir="ltr"><<a href="mailto:shrinand@maginatics.com" target="_blank">shrinand@maginatics.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hi,<br><div class="gmail_quote"><div dir="ltr"><div><br></div><div>There have been several articles which talk about keeping the number of objects in a container to about 1M. Beyond that sqlite starts becoming the bottleneck. I am going to make sure we abide by this number.</div>
<div><br></div><div>However, has anyone measured whether putting objects among multiple containers right from the start gives any performance benefits. For e.g. I could create 32 containers right at the start and split the objects among these as I write more and more objects. In the average case, I would have several partially filled containers instead of a few fully filled ones (fully filled means having 1M objects). Would this be better for the overall performance? Any downsides of this approach? Has anyone tried this before and published numbers on this?</div>
<div><br></div><div>Thanks in advance.</div><div>-Shri</div><div><br></div></div>
</div><br></div>
<br></div></div>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>