Kubelet启动报错Failed to start ContainerManager failed to initialize top level QOS containers

最近往K8s集群中添加节点的时候,发现部分节点的kubelet进程无法启动,导致节点处于NotReady状态。journalctl -u kubelet查看日志可以发现类似的日志:

....
Nov 29 23:32:13 localhost kubelet[3830]: I1129 23:32:13.311881    3830 server.go:333] Adding debug handlers to kubelet server.
Nov 29 23:32:13 localhost kubelet[3830]: W1129 23:32:13.311922    3830 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d
Nov 29 23:32:13 localhost kubelet[3830]: E1129 23:32:13.312092    3830 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Nov 29 23:32:13 localhost kubelet[3830]: I1129 23:32:13.382130    3830 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Nov 29 23:32:13 localhost kubelet[3830]: I1129 23:32:13.383695    3830 cpu_manager.go:155] [cpumanager] starting with none policy
Nov 29 23:32:13 localhost kubelet[3830]: I1129 23:32:13.383705    3830 cpu_manager.go:156] [cpumanager] reconciling every 10s
Nov 29 23:32:13 localhost kubelet[3830]: I1129 23:32:13.383713    3830 policy_none.go:42] [cpumanager] none policy: Start
Nov 29 23:32:13 localhost kubelet[3830]: F1129 23:32:13.384272    3830 kubelet.go:1384] Failed to start ContainerManager failed to initialize top level QOS containers: failed to update top level BestEffort QOS cgroup : failed to set supported cgroup subsystems for cgroup [kubepods besteffort]: Failed to set config for supported subsystems : failed to write 4611686018427387904 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-besteffort.slice/hugetlb.1GB.limit_in_bytes: no such file or directory
Nov 29 23:32:13 localhost systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a

日志里的Failed to start ContainerManager failed to initialize top level QOS containers: failed to update top level BestEffort QOS cgroup : failed to set supported cgroup subsystems for cgroup [kubepods besteffort]: Failed to set config for supported subsystems : failed to write 4611686018427387904 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-besteffort.slice/hugetlb.1GB.limit_in_bytes: no such file or directory就是问题所在,查了一下,确实这个目录不存在,很奇怪,这个subsystem明明就是kubelet所创建的啊。

之前的做法就是重启一下这个节点,重启完成问题也解决了,不过显然重启不是最优解,于是还是动手查了一下,发现了这个issue: Failed to start ContainerManager failed to initialise top level QOS containers #43856,虽然这个issue已经被修复关闭了,但实际我们还是遇到了类似的问题,好在评论里给出了一种解决办法,执行:

for i in $(systemctl list-unit-files --no-legend --no-pager -l | grep --color=never -o .*.slice | grep kubepod);
do systemctl stop $i;
done

原理也很简单,就是停掉kubepod相关的systemd slice,经过测试,是可以解决问题的,简单点的方法,直接执行systemctl stop kubepods.slice后,再重启kubelet,问题也就解决了。

具体的深层次原因没有深究,可能的情况是默认kubelet依赖kubepods.slicekubepods-besteffort.slicekubepods-burstable.slice这三个subsystem,但是因为某些原因这三个并没有完整启动,导致启动kubelet失败,停止kubepods.slice之后,kubelet尝试重启所有的subsytem,问题解决。