OpenstackSwift - rclone Copy/Sync Large Objects set (150Mil objects - 80TB) from Tenant PROD1 to PROD2
prem shivakumar
psk.9009 at gmail.com
Fri Sep 25 21:47:47 UTC 2020
Hi Open-stack users,
Our current setup:
PROD1
Container1
Dir1/Dir2/YYYYMMDD.tar
Want to move to:
PROD2
Container2_YYYYMM
Dir1/Dir2/YYYYMMDD.tar
I got all the objects metadata(Object names) using “swift list Container1”
I am currently using rclone where I am passing metadata(Object names) info
of “Container1”.
All the Auth variables exist in rclone.conf.
Example command:
rclone copy PROD1: Dir1/Dir2/YYYYMMDD.tar PROD2:Container2_YYYYMM/Dir1/Dir2/
YYYYMMDD.tar
Currently, I am facing slowness in copying files from PROD1 to PROD2.
On an average, It takes about 5s to copy the object from PROD1 to PROD2.
If I count the time against 150M its more than 10000Days.
I have also tried “rclone sync with —fast-list” it is failing as it is
unable to keep the listing in the memory.
In this case, is there any easier and faster solution to copy data from
Container1 to Container2?
Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200925/28e07179/attachment.html>
More information about the openstack-discuss
mailing list