实战

回到首页 个人中心 反馈问题 注册登录
下载APP
首页 课程 实战 体系课 手记 专栏 慕课教程

纯正的worker节点（对应课程node-3）启动后报错：kubelet.go:2243] node "emon3" not found

来源：5-6 部署kubernetes工作节点

夜愿小夜

2022-04-01

4月 01 14:37:32 emon3 systemd[1]: Started Kubernetes Kubelet.
4月 01 14:37:32 emon3 kubelet[61448]: I0401 14:37:32.405844   61448 flags.go:59] FLAG: --add-dir-header="false"
4月 01 14:37:32 emon3 kubelet[61448]: I0401 14:37:32.406136   61448 flags.go:59] FLAG: --address="0.0.0.0"
。。。。。。省略一部分FLAG输出，慕课网提交报错。。。。。。

4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.779177   61448 plugins.go:635] Loaded volume plugin "kubernetes.io/csi"
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.779448   61448 server.go:1176] Started kubelet
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.780029   61448 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"emon3.16e1b25e8e5c7b9f", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"emon3", UID:"emon3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"Starting", Message:"Starting kubelet.", Source:v1.EventSource{Component:"kubelet", Host:"emon3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc089c54c6e74d19f, ext:5445898336, loc:(*time.Location)(0x70d1080)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc089c54c6e74d19f, ext:5445898336, loc:(*time.Location)(0x70d1080)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Post "https://127.0.0.1:6443/api/v1/namespaces/default/events": dial tcp 127.0.0.1:6443: connect: connection refused'(may retry after sleeping)
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.780185   61448 csi_plugin.go:1016] Failed to contact API server when waiting for CSINode publishing: Get "https://127.0.0.1:6443/apis/storage.k8s.io/v1/csinodes/emon3": dial tcp 127.0.0.1:6443: connect: connection refused
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.780211   61448 server.go:148] Starting to listen on 192.168.200.118:10250
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.780811   61448 server.go:410] Adding debug handlers to kubelet server.
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.782492   61448 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.787706   61448 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": unable to find data in memory cache.
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.787721   61448 kubelet.go:1274] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.787858   61448 volume_manager.go:269] The desired_state_of_world populator starts
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.787878   61448 volume_manager.go:271] Starting Kubelet Volume Manager
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.787999   61448 desired_state_of_world_populator.go:142] Desired state populator starts to run
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.788160   61448 reflector.go:219] Starting reflector *v1.CSIDriver (0s) from k8s.io/client-go/informers/factory.go:134
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.789004   61448 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://127.0.0.1:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.789438   61448 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/emon3?timeout=10s": dial tcp 127.0.0.1:6443: connect: connection refused
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.792448   61448 kubelet.go:2163] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.792601   61448 client.go:86] parsed scheme: "unix"
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.792622   61448 client.go:86] scheme "unix" not registered, fallback to default scheme
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.792710   61448 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.792719   61448 clientconn.go:948] ClientConn switching balancer to "pick_first"
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.792881   61448 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0002b63f0, {CONNECTING <nil>}
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.795942   61448 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0002b63f0, {READY <nil>}
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.797451   61448 factory.go:137] Registering containerd factory
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.797547   61448 factory.go:55] Registering systemd factory
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.797760   61448 factory.go:101] Registering Raw factory
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.797884   61448 manager.go:1203] Started watching for new ooms in manager
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.798398   61448 manager.go:301] Starting recovery of all containers
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.809186   61448 manager.go:306] Recovery completed
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.810920   61448 kubelet_network_linux.go:56] Initialized IPv4 iptables rules.
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.810943   61448 status_manager.go:158] Starting to sync pod status with apiserver
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.810957   61448 kubelet.go:1802] Starting kubelet main sync loop.
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.811113   61448 kubelet.go:1826] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.812350   61448 reflector.go:219] Starting reflector *v1.RuntimeClass (0s) from k8s.io/client-go/informers/factory.go:134
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.828432   61448 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https://127.0.0.1:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp 127.0.0.1:6443: connect: connection refused
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.872623   61448 kubelet_node_status.go:339] Setting node annotation to enable volume controller attach/detach
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.873316   61448 setters.go:86] Using node IP: "192.168.200.118"
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.874053   61448 kubelet_node_status.go:531] Recording NodeHasSufficientMemory event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.874071   61448 kubelet_node_status.go:531] Recording NodeHasNoDiskPressure event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.874076   61448 kubelet_node_status.go:531] Recording NodeHasSufficientPID event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.875350   61448 cpu_manager.go:193] [cpumanager] starting with none policy
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.875379   61448 cpu_manager.go:194] [cpumanager] reconciling every 10s
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.875397   61448 state_mem.go:36] [cpumanager] initializing new in-memory state store
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.875581   61448 state_mem.go:88] [cpumanager] updated default cpuset: ""
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.875590   61448 state_mem.go:96] [cpumanager] updated cpuset assignments: "map[]"
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.875637   61448 state_checkpoint.go:136] [cpumanager] state checkpoint: restored state from checkpoint
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.875644   61448 state_checkpoint.go:137] [cpumanager] state checkpoint: defaultCPUSet:
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.875649   61448 policy_none.go:43] [cpumanager] none policy: Start
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.881551   61448 manager.go:236] Starting Device Plugin manager
4月 01 14:37:37 emon3 kubelet[61448]: W0401 14:37:37.881593   61448 manager.go:594] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.884735   61448 manager.go:278] Serving device plugin registration server on "/var/lib/kubelet/device-plugins/kubelet.sock"
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.884876   61448 plugin_watcher.go:52] Plugin Watcher Start at /var/lib/kubelet/plugins_registry
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.884962   61448 plugin_manager.go:112] The desired_state_of_world populator (plugin watcher) starts
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.884969   61448 plugin_manager.go:114] Starting Kubelet Plugin Manager
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.885777   61448 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get node info: node "emon3" not found
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.888074   61448 kubelet_node_status.go:339] Setting node annotation to enable volume controller attach/detach
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.888248   61448 setters.go:86] Using node IP: "192.168.200.118"
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.889301   61448 kubelet.go:2243] node "emon3" not found
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.889427   61448 kubelet_node_status.go:531] Recording NodeHasSufficientMemory event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.889440   61448 kubelet_node_status.go:531] Recording NodeHasNoDiskPressure event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.889445   61448 kubelet_node_status.go:531] Recording NodeHasSufficientPID event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.889459   61448 kubelet_node_status.go:71] Attempting to register node emon3
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.889686   61448 kubelet_node_status.go:93] Unable to register node "emon3" with API server: Post "https://127.0.0.1:6443/api/v1/nodes": dial tcp 127.0.0.1:6443: connect: connection refused
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.911249   61448 kubelet.go:1888] SyncLoop (ADD, "file"): "nginx-proxy-emon3_kube-system(7cf27d45a874e9a25860392cd83c3e6a)"
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.911297   61448 topology_manager.go:187] [topologymanager] Topology Admit Handler
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.911387   61448 kubelet_node_status.go:339] Setting node annotation to enable volume controller attach/detach
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.911624   61448 setters.go:86] Using node IP: "192.168.200.118"
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.912570   61448 kubelet_node_status.go:531] Recording NodeHasSufficientMemory event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.912600   61448 kubelet_node_status.go:531] Recording NodeHasNoDiskPressure event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.912608   61448 kubelet_node_status.go:531] Recording NodeHasSufficientPID event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.912798   61448 kubelet_node_status.go:339] Setting node annotation to enable volume controller attach/detach
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.912938   61448 setters.go:86] Using node IP: "192.168.200.118"
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.913333   61448 kubelet_node_status.go:531] Recording NodeHasSufficientMemory event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.913344   61448 kubelet_node_status.go:531] Recording NodeHasNoDiskPressure event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: I0401 14:37:37.913349   61448 kubelet_node_status.go:531] Recording NodeHasSufficientPID event message for node emon3
4月 01 14:37:37 emon3 kubelet[61448]: W0401 14:37:37.913803   61448 status_manager.go:550] Failed to get status for pod "nginx-proxy-emon3_kube-system(7cf27d45a874e9a25860392cd83c3e6a)": Get "https://127.0.0.1:6443/api/v1/namespaces/kube-system/pods/nginx-proxy-emon3": dial tcp 127.0.0.1:6443: connect: connection refused
4月 01 14:37:37 emon3 kubelet[61448]: E0401 14:37:37.989343   61448 kubelet.go:2243] node "emon3" not found

crictl看不到镜像！等了半天了。该问题已卡主2天，二进制安装走不下去了！！！

$ crictl images
IMAGE               TAG                 IMAGE ID            SIZE


$ ctr plugin ls|grep error
io.containerd.snapshotter.v1    aufs                     linux/amd64    error     
io.containerd.snapshotter.v1    btrfs                    linux/amd64    error     
io.containerd.snapshotter.v1    devmapper                linux/amd64    error     
io.containerd.snapshotter.v1    zfs                      linux/amd64    error

$ crictl ps -a
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID

一切的一切，都是按照老师的文档来的，虚拟机还原，重新安装了一遍，还是不行；
我认为这个是很大很大的问题，老师给的思路是，看kubelet加载的nginx，然后看 journal -l -u kublet日志，看了，实在是搞不定。

非常非常期望，帮助解决一下！！！
在线等，目前环境还在，需要提供什么内容，会随时配合！！！

写回答

2回答

夜愿小夜

提问者

2022-04-01

自我回答一下！！！

问题一直没决绝比较烦躁，打算从头再来一遍！

发现了一个问题，我的机器CentOS7.7版本：

$ uname -a

Linux emon3 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release 

CentOS Linux release 7.9.2009 (Core)

在执行  sysctl -p /etc/sysctl.d/kubernetes.conf 时发生了错误：

sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: 没有那个文件或目录

后来解决了，通过 modprobe br_netfilter

但该方式是临时的，重启失效！

那就临时+永久！

解决临时：

modprobe br_netfilter

解决永久（不含临时生效）：cat > /etc/sysconfig/modules/br_netfilter.modules << EOFmodprobe br_netfilterEOF

$ cat > /etc/sysconfig/modules/br_netfilter.modules << EOFmodprobe br_netfilterEOF

$ chmod 755 /etc/sysconfig/modules/br_netfilter.modules

# 查看下
$ lsmod |grep br_netfilter
br_netfilter           22256  0 
bridge                151336  1 br_netfilter

解决了该问题，继续安装！

等到worker解决部署时心里打鼓，不确定，然后果然还是碰到了

kubelet.go:2243] node "node-3" not found

这个大大的问题！心态爆炸！！！

想着继续看视频吧，至少不还有 node-2 节点可用嘛，先继续！！！

等了约莫10分钟后，又在node-3几点查看了下：

4月 01 17:27:56 emon3 kubelet[60611]: E0401 17:27:56.859842   60611 kubelet.go:2243] node "emon3" not found
4月 01 17:27:56 emon3 kubelet[60611]: E0401 17:27:56.959939   60611 kubelet.go:2243] node "emon3" not found
4月 01 17:27:57 emon3 kubelet[60611]: E0401 17:27:57.060894   60611 kubelet.go:2243] node "emon3" not found
4月 01 17:27:59 emon3 kubelet[60611]: I0401 17:27:59.174140   60611 setters.go:86] Using node IP: "192.168.200.118"
4月 01 17:28:00 emon3 kubelet[60611]: E0401 17:28:00.488025   60611 kubelet.go:2163] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
4月 01 17:28:05 emon3 kubelet[60611]: I0401 17:28:05.318879   60611 kubelet_getters.go:176] "Pod status updated" pod="kube-system/nginx-proxy-emon3" status=Running
4月 01 17:28:05 emon3 kubelet[60611]: E0401 17:28:05.489211   60611 kubelet.go:2163] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
4月 01 17:28:09 emon3 kubelet[60611]: I0401 17:28:09.178972   60611 setters.go:86] Using node IP: "192.168.200.118"
4月 01 17:28:10 emon3 kubelet[60611]: E0401 17:28:10.490192   60611 kubelet.go:2163] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
4月 01 17:28:15 emon3 kubelet[60611]: E0401 17:28:15.491678   60611 kubelet.go:2163] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
4月 01 17:28:19 emon3 kubelet[60611]: I0401 17:28:19.182519   60611 setters.go:86] Using node IP: "192.168.200.118"
4月 01 17:28:20 emon3 kubelet[60611]: E0401 17:28:20.493213   60611 kubelet.go:2163] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

发现，好了，然后查看容器，启动了！

$ crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
ed9cebb7f2bd2       f0b8a9a541369       30 minutes ago      Running             nginx-proxy         0                   930487ad733e3

这感觉真是酸爽！

好了，以上是我的问题解决办法，希望对后来学习者有用。

截止目前，我也闹不清是 sysctl -p /etc/sysctl.d/kubernetes.conf  问题，还是我等待时间不够，也可能两者都有！

学习，需要认真细心，也需要耐心，更需要总结归纳！希望大家都学好，提升自己，更上一层楼！！！

2
2
夜愿小夜
回复
qq_imba的bug_zBIZ29
在做了配置的，在配置正确情况下，也需要等待一段时间。仔细看下我的描述，希望能解决你的问题。
2023-01-16
共2条回复

Mr_Lee_Bin

2022-08-11

me解决方案：  
#先下载 ctr pull registry.cn-hangzhou.aliyuncs.com/imooc2/redis:alpine
#再测试 ctr run -t -d registry.cn-hangzhou.aliyuncs.com/imooc2/redis:alpine redis 跑不起来，
#卸载rpm -e libseccomp-devel-2.3.1-4.el7.x86_64 --nodeps，rpm -e libseccomp-2.3.1-4.el7.x86_64 --nodeps
#下载高于2.4以上的包wget http://rpmfind.net/linux/centos/8-stream/BaseOS/x86_64/os/Packages/libseccomp-2.5.1-1.el8.x86_64.rpm
#安装rpm -ivh libseccomp-2.5.1-1.el8.x86_64.rpm再run,OK! 
#回看node3    journalctl -f -u kubelet 和journalctl -f -u kube-proxy好了。


0
0

Kubernetes生产落地全程实践

一个互联网公司落地Kubernetes全过程点点滴滴

2293 学习 · 2216 问题

查看课程

相似问题

【已解决】node3工作节点node "k8s-node3" not found

回答 3

部署es报错

回答 2

【已解决】containerd 拉取 hub.mooc.com镜像报错

回答 1

node "node-2" not found

回答 1

启动kubelet的报错

回答 2

打开慕课网App查看更多内容