Open Stack

For those playing along from home, this question has been discussed at https://answers.launchpad.net/swift/+question/233444

--John


On Aug 3, 2013, at 10:34 AM, kalrey <wuse.kalrey at gmail.com> wrote:

> hi openstackers，
> I'm a learner of swift. I took some benchmark about swift last week and the result is not pleasant.
> When I put a large number of small files(4KB) under high concurrency, the performance degradation  of PUT appeared.
> The speed of PUT even can reach to 2000/s at beginning. But it down to 600/s after one minute. It's stable at 100/s at last and some error like '503' occured. But when I flushed all disk in cluster it could reach back 2000/s.
> In fact, I also took some benchmark about GET in the same environment but it works very well(5000/s).
>  
> There are some information which maybe useful:
> Test environment:
> Ubuntu 12.04
> 1 proxy-node : 128GB-ram / CPU 16core / 1Gb NIC*1
> 5 Storage-nodes : each for 128GB-ram / CPU 16core / 2TB*4 / 1Gb NIC*1.
> [bench]
> 
> concurrency 
> = 200
> 
> object_size 
> = 4096
> 
> num_objects 
> = 2000000
> 
> num_containers 
> = 200
> =============================================
> I have traced the code of PUT operation to find out what cause the performance degradation while putting objects. Some code cost a long time in ObjectController::PUT(swift/obj/server.py).
>  
> > for chunk in iter(lambda: reader(self.network_chunk_size), ”):
>             start_time = time.time()
> >          upload_size += len(chunk)
> >          if time.time() > upload_expiration:
> >                  self.logger.increment(‘PUT.timeouts’)
> >                  return HTTPRequestTimeout(request=request)
> >          etag.update(chunk)
> >          while chunk:
> >                 written = os.write(fd, chunk)
> >                 chunk = chunk[written:]
> >          sleep() 
>  
>        'lambda: reader' will cost average of 600ms per execution. And 'sleep()' will cost 500ms per execution.In fact, 'fsync' also spend a lot time when file flush to disk at last and I removed it already just for testing. I think the time is too long.
> I monitor resource of cluster while putting object.The usage of bandwidth is very low and the load of CPUs were very light.
> I have tried to change vfs_cache_pressure to a low value and it does not seem to work.  
> Is there any advice to figure out the problem?
> appreciate~
> kalrey
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130803/9438b474/attachment.pgp>