The CloudBeaver ECS tasks started crash-looping yesterday. We spent a few hours debugging an intermittent "password authentication failed" error in CloudBeaver running on ECS with an RDS PostgreSQL backend. The root cause turned out to be an undocumented interaction between CloudBeaver's config parser and special characters in auto-generated passwords. Here's the full investigation. The Symptom … Continue reading CloudBeaver on ECS: Why Password Authentication Fails Intermittently
Tag: RDS
Use AWS Secret Manager to handle credentials
AWS Secret Manager is a great solution for secret management. It is similar to HarshiCorp Vault, but with better integrations with other AWS services, e.g. IAM, RDS, Redshift, DocumentDB. As illustrated above, I created a database in RDS and a credential in Secret Manager, then attached the credential to the database for dynamic reference. The … Continue reading Use AWS Secret Manager to handle credentials
CloudFormation takes 8 hours to complete
I used CloudFormation to restore a RDS snapshot to a new instance, and it took 8 hours to complete!! The original instance (where the snapshot was taken) disk size is 45G, and the new instance is 50G which I specified in the CloudFormation template. According to AWS, that is the reason why it took so … Continue reading CloudFormation takes 8 hours to complete
Check DNS Cache when deploy Multi-AZ RDS
It is a common practice to enable mutli-AZ when deploy RDS in AWS to provide high availability. That's not all. There is one more thing you need to check - DNS cache setting of your applications/codes. If you overlook it, the hight availability design may not work as you expect. Imagine that when RDS detects the failure … Continue reading Check DNS Cache when deploy Multi-AZ RDS
