As mentioned in my earlier posts, we use
aws s3 sync to migrate a large number of files from on-premise to AWS. Here are a few things that I learned:
- The AWS credential can be renewed by an external process for a long running AWS CLI process. (Renew AWS credential for a long run AWS CLI process)
- Some parameters can be tuned to decrease or increase the transfer speed, e.g
max_concurrent_requests. (AWS S3 CLI configuration)
--quietoption suppresses the ExpireToken error. You won’t see the following error if it is used.
An error occurred (ExpiredToken) when calling the PutObject operation: The provided token has expired.
--exact-timestampsoption only works when syncing from S3 to local, same-sized items will be ignored only when the timestamps match exactly. The default behavior is to ignore same-sized items unless the local version is newer than the S3 version. (Source code)
One thought on “`aws s3 sync` lessons learned”