I used rsync to transfer data, but it’s too slow. I’m looking for alternative methods.
By enabling the ‘Allow external connections’ setting on your bucket and using S3 tools like rclone or Cyberduck, you can upload directly to DataHub for faster uploads.
When uploading a directory containing multiple files instead of a single 320GB file, you can upload more quickly by using rclone to perform parallel uploads.
rclone copy /source/path your-datahub-config:bucket-name --transfers=8 --s3-no-check-bucket --inplace