OpenstackSwift - rclone Copy/Sync Large Objects set (150Mil objects - 80TB) from Tenant PROD1 to PROD2
Hi Open-stack users, Our current setup: PROD1 Container1 Dir1/Dir2/YYYYMMDD.tar Want to move to: PROD2 Container2_YYYYMM Dir1/Dir2/YYYYMMDD.tar I got all the objects metadata(Object names) using “swift list Container1” I am currently using rclone where I am passing metadata(Object names) info of “Container1”. All the Auth variables exist in rclone.conf. Example command: rclone copy PROD1: Dir1/Dir2/YYYYMMDD.tar PROD2:Container2_YYYYMM/Dir1/Dir2/ YYYYMMDD.tar Currently, I am facing slowness in copying files from PROD1 to PROD2. On an average, It takes about 5s to copy the object from PROD1 to PROD2. If I count the time against 150M its more than 10000Days. I have also tried “rclone sync with —fast-list” it is failing as it is unable to keep the listing in the memory. In this case, is there any easier and faster solution to copy data from Container1 to Container2? Thank you.
participants (1)
-
prem shivakumar