Open Stack

Chuck made some interesting suggestions on the Openstack-Swift irc
channel related to this topic. Noting them down her in case this
benefits others.

1. Changing the partition power of the object server rings. I was
running with a partition power of 8. I ran some experiments changing
this to 4 and then even 0. Turned out that this actually hurt
performance. I confirmed that the number of directories created in the
objects directory was 2 ^ partition power. I guess with reduced number
of directories (in the case of 0, there was just one directory) there
was more contention at the xfs file-system causing the system to give
lower throughput.

I tested with partition powers of 0, 4, 7, 8, 16. 7 because the swift
ring calculator suggested so [1]. I got the best throughput at 8
rings.

2. Tweaking object_chunk_size_, client_chunk_size. This is 64K by
default and I was also always expecting to use 64K objects. Therefore
I didn't experiment with that.

3. Also, since this VM is entirely dedicated to Swift, I changed the

# disable TIME_WAIT...
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1

# disable syn cookies
net.ipv4.tcp_syncookies = 0

I was initially testing on a "not-so-enterprise-standard hardwar". I
changed to an enterprise-grade system. The VM was running on an
appropriately provisioned ESX host with enough memory, CPU and SSD.
The VM had 8 cores and 8GB memory. I was able to get a throughput of
~40MB/s (640 objects/second of 64KB each) without sharding. With
sharding objects across containers, I am getting ~50MB/s (800
objects/second of 64KB each).

-Shri

[1] http://rackerlabs.github.io/swift-ppc/

On Mon, Mar 3, 2014 at 10:36 AM, Shrinand Javadekar
<shrinand at maginatics.com> wrote:
> One of the options that a colleague of mine came up with is related to
> the on disk directory structure of the objects. I believe Swift hashes
> every object, uses the trailing N bytes of the hash itself to name the
> directory into which this object is put. Is this configurable in any
> way? For e.g. can I try and put many objects into a single directory
> by making N = 1. This will reduce the amount of work done when a
> single object is written.
>
> What do you think?
> -Shri
>
> On Sat, Mar 1, 2014 at 2:25 PM, Shrinand Javadekar
> <shrinand at maginatics.com> wrote:
>> Hi,
>>
>> I have single node Swift instance running in a VM. It has: 4 cores, 16
>> GB memory and 300GB SSD disk
>>
>> I want to get the best possible throughput from this Swift instance
>> when, say 100 clients are writing data concurrently. Are there any
>> recommendations to achieve this?
>>
>> So far, I've tried the following:
>>
>> 1. I tweaked the # of workers threads in the config files and it seems
>> that I get best results when I have:
>>
>> # grep workers *.conf
>> account-server.conf:workers = 16
>> container-server.conf:workers = 16
>> object-server.conf:workers = 64
>> proxy-server.conf:workers = 64
>>
>> 2. I am fine with not using SSL.
>> 3. Sharding data across multiple containers also gives me better results.
>>
>> I think I also observed better throughput with larger objects, but the
>> object size I need to work with cannot be changed. I have to 64K
>> objects.
>>
>> With these options, the best I got was ~20MB/s, whereas a simple dd
>> shows that the disk is capable of ~128MB/s (1Gbps).
>>
>> Are there other options that I can tweak for getting better throughput?
>>
>> Thanks in advance.
>> -Shri