静态pod nginx无法启动

来源：5-6 部署kubernetes工作节点

慕UI0500174

2022-03-31

containerd config default > /etc/containerd/config.toml
上面命令生成的默认配置文件在我的服务器上无法启动，然后改了一个参数，如下：

snapshotter = "overlayfs"
改为
snapshotter = "native"

改完后可以正常启动containerd了

不过后面启动kubelet的时候遇到问题了，报错如下：

Mar 31 22:11:54 worker-1 kubelet[25483]: E0331 22:11:54.149048   25483 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/worker-1?timeout=10s": dial tcp 127.0.0.1:6443: connect: connection refused
Mar 31 22:11:54 worker-1 kubelet[25483]: W0331 22:11:54.149135   25483 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1.CSIDriver: Get "https://127.0.0.1:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
Mar 31 22:11:54 worker-1 kubelet[25483]: E0331 22:11:54.149195   25483 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://127.0.0.1:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
Mar 31 22:11:54 worker-1 kubelet[25483]: E0331 22:11:54.149743   25483 cri_stats_provider.go:455] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.native"
Mar 31 22:11:54 worker-1 kubelet[25483]: E0331 22:11:54.149783   25483 kubelet.go:1351] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Mar 31 22:11:54 worker-1 kubelet[25483]: E0331 22:11:54.151162   25483 kubelet.go:2347] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.153907   25483 factory.go:145] Registering containerd factory
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.154178   25483 factory.go:55] Registering systemd factory
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.154512   25483 factory.go:103] Registering Raw factory
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.154720   25483 manager.go:1203] Started watching for new ooms in manager
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.163738   25483 manager.go:304] Starting recovery of all containers
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.184572   25483 manager.go:309] Recovery completed
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.206865   25483 kubelet_network_linux.go:57] "Initialized protocol iptables rules." protocol=IPv4
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.236895   25483 kubelet_network_linux.go:57] "Initialized protocol iptables rules." protocol=IPv6
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.237080   25483 status_manager.go:159] "Starting to sync pod status with apiserver"
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.237176   25483 kubelet.go:1977] "Starting kubelet main sync loop"
Mar 31 22:11:54 worker-1 kubelet[25483]: E0331 22:11:54.237347   25483 kubelet.go:2001] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Mar 31 22:11:54 worker-1 kubelet[25483]: W0331 22:11:54.238731   25483 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1.RuntimeClass: Get "https://127.0.0.1:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
Mar 31 22:11:54 worker-1 kubelet[25483]: E0331 22:11:54.238795   25483 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https://127.0.0.1:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.246527   25483 kubelet_node_status.go:352] "Setting node annotation to enable volume controller attach/detach"
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.248794   25483 kubelet_node_status.go:563] "Recording event message for node" node="worker-1" event="NodeHasSufficientMemory"
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.248866   25483 kubelet_node_status.go:563] "Recording event message for node" node="worker-1" event="NodeHasNoDiskPressure"
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.248900   25483 kubelet_node_status.go:563] "Recording event message for node" node="worker-1" event="NodeHasSufficientPID"
Mar 31 22:11:54 worker-1 kubelet[25483]: I0331 22:11:54.248995   25483 kubelet_node_status.go:70] "Attempting to register node" node="worker-1"
Mar 31 22:11:54 worker-1 kubelet[25483]: E0331 22:11:54.249176   25483 kubelet.go:2422] "Error getting node" err="node \"worker-1\" not found"

cri_stats_provider.go:455] “Failed to get the info of the filesystem with mountpoint” err=“unable to find data in memory cache” mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.native"
上面这个报错应该是我改的native导致的老师这个不知道怎么改了。。

写回答

2回答

慕UI0500174

提问者

2022-04-01

老师这是改回默认overlayfs的containerd启动日志

```

Apr 01 10:24:47 master-1 systemd[1]: Started containerd container runtime.

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.730274596+08:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.730309244+08:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.730338480+08:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.730385604+08:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.730551215+08:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.730711269+08:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.731630704+08:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.731677932+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.731827305+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.731890345+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.731924109+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.731948248+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.731976800+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.731998676+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.732027230+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.732055877+08:00" level=info msg="loading plugin \"io.container

d.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.732084150+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.732294128+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.732345245+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.732408778+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.732459302+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.733031448+08:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.733126314+08:00" level=info msg="Connect containerd service"

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.733245326+08:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to find snapshotter \"overlayfs\""

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.733286309+08:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.733808220+08:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.733920049+08:00" level=info msg=serving... address=/run/containerd/containerd.sock

Apr 01 10:24:47 master-1 containerd[3217]: time="2022-04-01T10:24:47.734051070+08:00" level=info msg="containerd successfully booted in 0.056999s"

```

========

以下是containerd的插件状态：

```

[root@master-1 ~]# ctr plugin ls

TYPE ID PLATFORMS STATUS

io.containerd.content.v1 content - ok

io.containerd.snapshotter.v1 aufs linux/amd64 error

io.containerd.snapshotter.v1 btrfs linux/amd64 error

io.containerd.snapshotter.v1 devmapper linux/amd64 error

io.containerd.snapshotter.v1 native linux/amd64 ok

io.containerd.snapshotter.v1 overlayfs linux/amd64 error

io.containerd.snapshotter.v1 zfs linux/amd64 error

io.containerd.metadata.v1 bolt - ok

io.containerd.differ.v1 walking linux/amd64 ok

io.containerd.gc.v1 scheduler - ok

io.containerd.service.v1 introspection-service - ok

io.containerd.service.v1 containers-service - ok

io.containerd.service.v1 content-service - ok

io.containerd.service.v1 diff-service - ok

io.containerd.service.v1 images-service - ok

io.containerd.service.v1 leases-service - ok

io.containerd.service.v1 namespaces-service - ok

io.containerd.service.v1 snapshots-service - ok

io.containerd.runtime.v1 linux linux/amd64 ok

io.containerd.runtime.v2 task linux/amd64 ok

io.containerd.monitor.v1 cgroups linux/amd64 ok

io.containerd.service.v1 tasks-service - ok

io.containerd.internal.v1 restart - ok

io.containerd.grpc.v1 containers - ok

io.containerd.grpc.v1 content - ok

io.containerd.grpc.v1 diff - ok

io.containerd.grpc.v1 events - ok

io.containerd.grpc.v1 healthcheck - ok

io.containerd.grpc.v1 images - ok

io.containerd.grpc.v1 leases - ok

io.containerd.grpc.v1 namespaces - ok

io.containerd.internal.v1 opt - ok

io.containerd.grpc.v1 snapshots - ok

io.containerd.grpc.v1 tasks - ok

io.containerd.grpc.v1 version - ok

io.containerd.grpc.v1 cri linux/amd64 error

[root@master-1 ~]# crictl ps

FATA[0000] listing containers: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

```

=======

以下是系统overlay模块状态和文件系统

```

[root@master-1 ~]# lsmod | grep overlay

overlay 91659 0

[root@master-1 ~]# cat /etc/fstab

# /etc/fstab

# Created by anaconda on Mon Sep 5 11:52:12 2016

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

UUID=f13d84b4-c756-4d89-9d5e-6b534397aa14 / xfs defaults 0 0

```

慕UI0500174

夜愿小夜

非常感谢老哥的回答问题解决了

2022-04-01

共3条回复

刘果国

2022-04-01

改回overlayfs吧，看看为啥containerd起不来。看看文件系统是不是有什么特殊的

慕UI0500174

给老师添麻烦了问题已解决

2022-04-01

共2条回复

Kubernetes生产落地全程实践

一个互联网公司落地Kubernetes全过程点点滴滴

2293 学习 · 2216 问题

查看课程

相似问题

ingress-nginx无法启动

回答 2

nginx启动问题

回答 1

nginx 容器是哪里创建的

回答 1

coredns 一个pod pending状态

回答 1

kubelet 无法启动在node3上

回答 3

打开慕课网App查看更多内容