Open Stack

Yep, you're right. Doing a HEAD request before every PUT gets expensive, especially for small files.

But don't despair! There's some good news.

First, realize that swiftclient is written for a pretty general use case. If you have more knowledge about how your system works, then you can write something more efficient. python-swiftclient has a CLI, yes, but it also provides a python module so that you can write your own client as needed. For example, you can write your own script that doesn't do HEAD requests before a PUT.

Second, this specific issue is being addressed in patch https://review.openstack.org/#/c/81646/ which adds If-None-Match support to PUT requests in Swift. This will allow a PUT without a HEAD and also avoid data transfer if the object already exists. I hope this patch lands today and is included in Icehouse.

--John





On Apr 2, 2014, at 7:52 AM, Roman Kravets <softded at gmail.com> wrote:

> Hello.
> 
> I have trouble with python-swiftclient during upload to match small
> file to Openstack Swift.
> 
> It client make HEAD request for each file during upload before PUT. It
> make very intencive read load to hard drive and as result all cluster
> show very small upload speed.
> 
> I see that for each file it make 6 HEAD request (3 recplicas * 2).
> 
> When I use cloudfuse this client make only PUT and as result hard
> drive load only to write data. It is make less IOps to hard drive.
> 
> What need make HEAD request for each files? How can I reduce the read
> request for reduce load to hard drive?
> 
> Thank you!
> 
> --
> Best regards,
> Roman Kravets
> 
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140402/0ee2feaf/attachment.sig>