Kubelet启动报错Failed to start ContainerManager failed to initialize top level QOS containers

最近往K8s集群中添加节点的时候，发现部分节点的kubelet进程无法启动，导致节点处于NotReady状态。journalctl -u kubelet查看日志可以发现类似的日志：

....
Nov 29 23:32:13 localhost kubelet[3830]: I1129 23:32:13.311881    3830 server.go:333] Adding debug handlers to kubelet server.
Nov 29 23:32:13 localhost kubelet[3830]: W1129 23:32:13.311922    3830 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d
Nov 29 23:32:13 localhost kubelet[3830]: E1129 23:32:13.312092    3830 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Nov 29 23:32:13 localhost kubelet[3830]: I1129 23:32:13.382130    3830 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Nov 29 23:32:13 localhost kubelet[3830]: I1129 23:32:13.383695    3830 cpu_manager.go:155] [cpumanager] starting with none policy
Nov 29 23:32:13 localhost kubelet[3830]: I1129 23:32:13.383705    3830 cpu_manager.go:156] [cpumanager] reconciling every 10s
Nov 29 23:32:13 localhost kubelet[3830]: I1129 23:32:13.383713    3830 policy_none.go:42] [cpumanager] none policy: Start
Nov 29 23:32:13 localhost kubelet[3830]: F1129 23:32:13.384272    3830 kubelet.go:1384] Failed to start ContainerManager failed to initialize top level QOS containers: failed to update top level BestEffort QOS cgroup : failed to set supported cgroup subsystems for cgroup [kubepods besteffort]: Failed to set config for supported subsystems : failed to write 4611686018427387904 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-besteffort.slice/hugetlb.1GB.limit_in_bytes: no such file or directory
Nov 29 23:32:13 localhost systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a

日志里的Failed to start ContainerManager failed to initialize top level QOS containers: failed to update top level BestEffort QOS cgroup : failed to set supported cgroup subsystems for cgroup [kubepods besteffort]: Failed to set config for supported subsystems : failed to write 4611686018427387904 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-besteffort.slice/hugetlb.1GB.limit_in_bytes: no such file or directory就是问题所在，查了一下，确实这个目录不存在，很奇怪，这个subsystem明明就是kubelet所创建的啊。

之前的做法就是重启一下这个节点，重启完成问题也解决了，不过显然重启不是最优解，于是还是动手查了一下，发现了这个issue： Failed to start ContainerManager failed to initialise top level QOS containers #43856，虽然这个issue已经被修复关闭了，但实际我们还是遇到了类似的问题，好在评论里给出了一种解决办法，执行：

for i in $(systemctl list-unit-files --no-legend --no-pager -l | grep --color=never -o .*.slice | grep kubepod);
do systemctl stop $i;
done

原理也很简单，就是停掉kubepod相关的systemd slice，经过测试，是可以解决问题的，简单点的方法，直接执行systemctl stop kubepods.slice后，再重启kubelet，问题也就解决了。

具体的深层次原因没有深究，可能的情况是默认kubelet依赖kubepods.slice、kubepods-besteffort.slice、kubepods-burstable.slice这三个subsystem，但是因为某些原因这三个并没有完整启动，导致启动kubelet失败，停止kubepods.slice之后，kubelet尝试重启所有的subsytem，问题解决。