Renew AWS credential for a long run AWS CLI process

We use aws s3 sync to synchronise a big mount of files (800,000+ objects) from on-premise to AWS S3 bucket. Due to security restrictions, the Maximum CLI/API session duration is configured for 1 hour. So it is most likely the credential will expire before the sync job is completed.

There are generally two places to configure the AWS credential: Environment variables or AWS credential config file.

  • Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY & AWS_SESSION_TOKEN) is not a solution here, as it is considered to be permanent and not looked upon again until end of the command.
  • Credential config file (e.g ~/.aws/credential) can be updated anytime, but the running AWS CLI won’t automatically re-read this file if the credential file is configured in the following format:
aws_access_key_id=AKIAIOSFODNN7EXAMPLE aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Here is a solution we implemented and it works like a charm!

  1. Create a codebuild project or lambda function in AWS, which is scheduled to run every 50 minutes to assume a IAM role (that is only allowed to put objects into the target S3 bucket), then saves the temp credential on the remote on-premise (e.g ~/.aws/temp.json) server in following format.
  "Version": 1,
  "AccessKeyId": "an AWS access key",
  "SecretAccessKey": "your AWS secret access key",
  "SessionToken": "the AWS session token for temporary credentials", 
  "Expiration": "ISO8601 timestamp when the credentials expire"

2. Update the AWS credential file (e.g ~/.aws/credential) on the on-premise server to

credential_process = cat ~/.aws/temp.json

In this way, AWS CLI will be smart enough to automatically renew the AWS credential that it is currently using by rerunning the credential_process command before they expire which timestamp is recorded int the Expiration key

Sourcing credentials with an external process


2 thoughts on “Renew AWS credential for a long run AWS CLI process

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s