Pods stuck in ContainerCreating with an event mentioning errno 524

Version/Environment (if relevant):

All versions on AWS/EKS

Issue:

Customers find Pods stuck in ContainerCreating and/or liveness probes fail with an event in the kubectl describe which mentions errno 524:

Warning FailedCreatePodSandBox 4m9s (x384 over 89m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524: unknown

Root Cause:

There is a linux kernel memory leak bug in the EKS optimized AMI based on linux kernel version 5.10.x. The linux bug can be verified by using cat /proc/sys/net/core/bpf_jit_limit and seeing what the limit currently is and by using cat /proc/vmallocinfo | grep bpf_jit | awk '{s+=$2} END {print s}', to see if the limit has been exceeded. Example:

[root@ip-10-0-35-23 /]# cat /proc/sys/net/core/bpf_jit_limit
264241152
[root@ip-10-0-35-23 /]# cat /proc/vmallocinfo | grep bpf_jit | awk '{s+=$2} END {print s}'
352337920

More details: https://github.com/awslabs/amazon-eks-ami/issues/1219#issuecomment-1533797378 “This issue is more likely to be encountered with kernel versions kernel-5.10.176-157.645.amzn2thrukernel-5.10.177-158.645.amzn where the rate of the memory leak is higher.”

Resolution:

AWS corrected the problem by patching by a new AMI Release AMI Release v20230501 awslabs/amazon-eks-ami, released May 3rd 2023. Temporary workaround is to run sysctl net.core.bpf_jit_limit=452534528 on all platform nodes by ssh’ing directly onto them. The permanent fix is to get your EKS to use this AMI: Release AMI Release v20230501 awslabs/amazon-eks-ami

Notes/Information:

Note the “temporary workaround” is short-term only, since the kernel bug can continue leaking.

​Version/Environment (if relevant):

​Issue:

​

​Root Cause:

​Resolution:

​Notes/Information:

Version/Environment (if relevant):

Issue:

Root Cause:

Resolution:

Notes/Information: