The myth of memory requests and limits in Kubernetes

I summarised a list of FAQ about the memory request and limit in Kubernetes. Also I have a real example down the bottom.

1) What if Container exceed memory request?
Totally fine, a Container can exceed its memory request if the Node has memory available.

2) What if Container exceed memory limit?
Thats dangerous, as a Container is not allowed to use more than its memory limit. If a Container allocates more memory than its limit, the Container becomes a candidate for termination. If the Container continues to consume memory beyond its limit, the Container is terminated. If a terminated Container is restartable, the kubelet will restart it, as with any other type of runtime failure.

3) What if you specify a Container’s memory limit, but not its memory request?
Container’s memory request is set to match its memory limit regardless if there is a default meomry request for the namespace.

4) What if you specify a Container’s memory request, but not its memory limit?
Container’s memory limit is set to the default memory limit for the namespace if there is one, otherwise it can use all of the available memory on the node.

5) What is the best practice of setting memory requests and limits?
By configuring memory requests and limits for the Containers that run in your cluster, you can make efficient use of the memory resources available on your cluster’s Nodes. By keeping a Pod’s memory request low, you give the Pod a good chance of being scheduled. By having a memory limit that is greater than the memory request, you accomplish two things:

• The Pod can have bursts of activity where it makes use of memory that happens to be available.
• The amount of memory a Pod can use during a burst is limited to some reasonable amount.

Here is a real case that I have seen in one of the kubernetes clusters: Users are unable to create new pods due to the error: ‘Failed Scheduling No nodes are available that match all of the following predicates:: Insufficient memory (4), MatchNodeSelector (3).’

In some cases, it could be caused by the old version of etcd (prior 3.0.17). For example:

https://github.com/kubernetes/kubernetes/issues/45237
https://github.com/kubernetes/kubernetes/issues/47150

If not caused by etcd, then most likely it is a real capacity related issue, run ‘kubectl describe nodes’ (Kubernetes) or  ‘oc describe nodes’ (OpenShift) to check the resources that have been allocated which looks like below:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ------------	----------	---------------	-------------
  100m (3%)	0 (0%)		14Gi (97%)	14Gi (97%)

How come Memory Requests == Memory Limits? We only configured container memory limit, no memory requests. Do you have the answer? If not, go back to #3 in the above list.

Reference:
https://kubernetes-v1-4.github.io/docs/user-guide/compute-resources/
https://blog.openshift.com/full-cluster-capacity-management-monitoring-openshift/
https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/
https://kubernetes.io/docs/tasks/administer-cluster/memory-default-namespace/