We recently had a Vault issue. It is Vault with HA, the data store is Consul. One day the Consul cluster went offline, and we noticed there are a few OOM errors in the logs. The servers have been running fine and no stress on the memory utilisation for quite a while. There must be a dramatic change somewhere.
After some analysis of the audit logs, it turns out that a developer uses Vault in a wrong (lazy) way – In stead of creating a token with TTL, and renew it or get a new one before the TTL expires. He just creates a new token each time when he needs to retrieve a credential from Vault, and set TTL to 0 which makes things worse. The consequence is that the keys stored in Consul increased 16 times within 3 days!! No wonder the servers are running out of memory.
