[Openstack] [OpenStack][Swift] Fast way of uploading 200GB of 200KB files to Swift

Robert van Leeuwen Robert.vanLeeuwen at spilgames.com
Mon Jan 14 14:27:43 UTC 2013


> I see. With replication switched off during upload, does inserting into various containers speed up the process
> or is it irrelevant?

I'm not sure what's your question but maybe this helps:
In short:
The replication daemon is "walking" across your files to check if any files need to be replicated to other nodes
(e.g. when a node was broken or a new node is added to the cluster).
Because it is scanning your filesystem it eats up io's. That is why it speeds up the system if you turn this off.

The number off objects in a container impacts the sqlite databases. Each container is a SQLite database.
The SQLite database will get slower with more objects in them.
I think the recommendations for spinning disks is max a few million for one container, when using SSD's for the databases you can go quite a bit higher though.

Cheers,
Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130114/ee81b222/attachment.html>


More information about the Openstack mailing list