When KMS key grant is created for Lambda execution role?

I was working on a new Lambda function a couple days ago. It took me a while to deploy it successfully via CloudFormation. It was all due to some required IAM policies were missed in the deployment role. So I had to delete the stack then create again for a couple of times.

When I tested the Lambda function, it complains that Lambda was unable to decrypt the environment variables because KMS access was denied. But I was 100% sure the Lambda execution role has the right permission to use the KMS key (aws/lambda) to decrypt.

It turns out that it was caused by the KMS key grant is missing. Normally
the grant to the KMS key for the Lambda execution role is created automatically when:

• a new function is created
• the execution role is changed for an existing function.
• the KMS key is changed for an existing function

Please note that the grant is given to the principle of the execution role, and the principle is unique. That is to say if the role is re-created, the principle will be different even the role ARN is still the same.

In my case, as I rebuilt the stack multiple times, and at some stage the deployment re-created the role but did not trigger new KMS grant for the new role.

The fix is simple if knowing the root cause. For me, I just changed the execution role to another one, then changed it back. In this way, it triggers Lambda to create a new grant.

Reference
https://forums.aws.amazon.com/thread.jspa?threadID=281599.