swarm
来源：7-5 集群服务间通信之Routing Mesh
yangxinxin
2019-11-29
我在swarm集群中创建service ：docker service create --name client --network demo -d busybox sh -c “while true; do sleep 36000; done”；总是运行在manager节点上，docker service ps client 看到该 service刚开始被分配到 worker2 ，然后又被分配到worker1，最后分配到manager，错误信息不知如何查看？？
写回答
1回答
-
yangxinxin
提问者
2019-11-29
当我在manager上启动一个service client，同时在work2上tail -f /var/log/messages，出现以下日志（只是其中的一部分），看样子似乎是network的原因：
Nov 29 17:08:15 localhost kernel: IPVS: Creating netns size=2048 id=145 Nov 29 17:08:15 localhost kernel: IPVS: Creating netns size=2048 id=146 Nov 29 17:08:15 localhost kernel: IPVS: Creating netns size=2048 id=147 Nov 29 17:08:15 localhost kernel: IPVS: Creating netns size=2048 id=148 Nov 29 17:08:16 localhost kernel: br0: port 1(vxlan0) entered blocking state Nov 29 17:08:16 localhost kernel: br0: port 1(vxlan0) entered disabled state Nov 29 17:08:16 localhost kernel: device vxlan0 entered promiscuous mode Nov 29 17:08:16 localhost kernel: br0: port 1(vxlan0) entered blocking state Nov 29 17:08:16 localhost kernel: br0: port 1(vxlan0) entered forwarding state Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered blocking state Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered disabled state Nov 29 17:08:16 localhost kernel: device veth0 entered promiscuous mode Nov 29 17:08:16 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered blocking state Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered forwarding state Nov 29 17:08:16 localhost kernel: br0: port 1(vxlan0) entered blocking state Nov 29 17:08:16 localhost kernel: br0: port 1(vxlan0) entered disabled state Nov 29 17:08:16 localhost kernel: device vxlan0 entered promiscuous mode Nov 29 17:08:16 localhost kernel: br0: port 1(vxlan0) entered blocking state Nov 29 17:08:16 localhost kernel: br0: port 1(vxlan0) entered forwarding state Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered blocking state Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered disabled state Nov 29 17:08:16 localhost kernel: device veth0 entered promiscuous mode Nov 29 17:08:16 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered blocking state Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered forwarding state Nov 29 17:08:16 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered disabled state Nov 29 17:08:16 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered blocking state Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered forwarding state Nov 29 17:08:16 localhost kernel: br0: port 2(veth0) entered disabled state Nov 29 17:08:16 localhost kernel: br0: port 1(vxlan0) entered disabled state Nov 29 17:08:16 localhost kernel: device veth0 left promiscuous mode Nov 29 17:08:16 localhost kernel: ov-001000-3697e: port 2(veth0) entered disabled state Nov 29 17:08:16 localhost kernel: device vxlan0 left promiscuous mode Nov 29 17:08:16 localhost kernel: ov-001000-3697e: port 1(vxlan0) entered disabled state Nov 29 17:08:16 localhost dockerd: time="2019-11-29T17:08:16.918102819+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint_count 3697eaa0ye8khxfed418b2z91], retrying...." Nov 29 17:08:16 localhost dockerd: time="2019-11-29T17:08:16.918978437+08:00" level=error msg="Failed creating ingress network: error creating external connectivity network: Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i docker_gwbridge -j RETURN: iptables: No chain/target/match by that name.\n (exit status 1))" Nov 29 17:08:16 localhost dockerd: time="2019-11-29T17:08:16.919037398+08:00" level=error msg="failed adding service binding for 75ef29f86a095ad6eca0e67d4f7f223586f3e41154b057e094bc49b5f2f6607d epRec:{whoami.1.a1mo4wp0lm6jz3wfcao9a3h0p whoami jshl46atjzwlitqars27znzy1 10.255.0.5 10.255.0.6 [&PortConfig{Name:,Protocol:TCP,TargetPort:8000,PublishedPort:8000,}] [] [13ceb7ebe29e] false} err:network 3697eaa0ye8khxfed418b2z91 not found" Nov 29 17:08:16 localhost dockerd: time="2019-11-29T17:08:16.919063707+08:00" level=error msg="failed adding container name resolution for eff3bd0c83a975679a4c12c1c04f45bc7c8402e6f9c924aa1ac66cff806e66ad epRec:{ingress-endpoint <nil> 10.255.0.2 [] [] [] false} err:network 3697eaa0ye8khxfed418b2z91 not found" Nov 29 17:08:17 localhost dockerd: time="2019-11-29T17:08:17.036904702+08:00" level=warning msg="rmServiceBinding handleEpTableEvent whoami 75ef29f86a095ad6eca0e67d4f7f223586f3e41154b057e094bc49b5f2f6607d aborted c.serviceBindings[skey] !ok" Nov 29 17:08:17 localhost dockerd: time="2019-11-29T17:08:17.036982773+08:00" level=error msg="failed removing container name resolution for eff3bd0c83a975679a4c12c1c04f45bc7c8402e6f9c924aa1ac66cff806e66ad epRec:{ingress-endpoint <nil> 10.255.0.2 [] [] [] false} err:network 3697eaa0ye8khxfed418b2z91 not found" Nov 29 17:08:20 localhost kernel: br0: port 3(veth1) entered blocking state Nov 29 17:08:20 localhost kernel: br0: port 3(veth1) entered disabled state Nov 29 17:08:20 localhost kernel: device veth1 entered promiscuous mode Nov 29 17:08:20 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): veth1: link is not ready Nov 29 17:08:20 localhost kernel: br0: port 3(veth1) entered disabled state Nov 29 17:08:20 localhost kernel: device veth1 left promiscuous mode Nov 29 17:08:20 localhost kernel: br0: port 3(veth1) entered disabled state Nov 29 17:08:20 localhost dockerd: time="2019-11-29T17:08:20.553996050+08:00" level=error msg="4d752674d89fbd66a1795905f5863cfce24946a48a6671f4c5627ac56207e001 cleanup: failed to delete container from containerd: no such container" Nov 29 17:08:20 localhost dockerd: time="2019-11-29T17:08:20.554033135+08:00" level=error msg="fatal task error" error="starting container failed: error creating external connectivity network: Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i docker_gwbridge -j RETURN: iptables: No chain/target/match by that name.\n (exit status 1))" module=node/agent/taskmanager node.id=mass33gxtedc70bl51go9gj5f service.id=0t2vjqboe61p9psmqpi5awan6 task.id=nwhrpdq14nmq0nnv2lv1p48l5 Nov 29 17:08:20 localhost kernel: IPVS: Creating netns size=2048 id=149 Nov 29 17:08:20 localhost kernel: IPVS: Creating netns size=2048 id=150 Nov 29 17:08:21 localhost kernel: br0: port 1(vxlan0) entered blocking state Nov 29 17:08:21 localhost kernel: br0: port 1(vxlan0) entered disabled state Nov 29 17:08:21 localhost kernel: device vxlan0 entered promiscuous mode Nov 29 17:08:21 localhost kernel: br0: port 1(vxlan0) entered blocking state Nov 29 17:08:21 localhost kernel: br0: port 1(vxlan0) entered forwarding state Nov 29 17:08:21 localhost kernel: br0: port 2(veth0) entered blocking state Nov 29 17:08:21 localhost kernel: br0: port 2(veth0) entered disabled state Nov 29 17:08:21 localhost kernel: device veth0 entered promiscuous mode Nov 29 17:08:21 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready Nov 29 17:08:21 localhost kernel: br0: port 2(veth0) entered blocking state Nov 29 17:08:21 localhost kernel: br0: port 2(veth0) entered forwarding state Nov 29 17:08:21 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready Nov 29 17:08:21 localhost kernel: br0: port 2(veth0) entered disabled state Nov 29 17:08:21 localhost kernel: br0: port 1(vxlan0) entered disabled state Nov 29 17:08:21 localhost kernel: device veth0 left promiscuous mode Nov 29 17:08:21 localhost kernel: ov-001000-3697e: port 2(veth0) entered disabled state Nov 29 17:08:21 localhost kernel: device vxlan0 left promiscuous mode Nov 29 17:08:21 localhost kernel: ov-001000-3697e: port 1(vxlan0) entered disabled state Nov 29 17:08:21 localhost dockerd: time="2019-11-29T17:08:21.414112713+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint_count 3697eaa0ye8khxfed418b2z91], retrying...." Nov 29 17:08:21 localhost dockerd: time="2019-11-29T17:08:21.414185823+08:00" level=error msg="Failed creating ingress network: error creating external connectivity network: Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i docker_gwbridge -j RETURN: iptables: No chain/target/match by that name.\n (exit status 1))" Nov 29 17:08:21 localhost dockerd: time="2019-11-29T17:08:21.414224939+08:00" level=error msg="failed adding service binding for 75ef29f86a095ad6eca0e67d4f7f223586f3e41154b057e094bc49b5f2f6607d epRec:{whoami.1.a1mo4wp0lm6jz3wfcao9a3h0p whoami jshl46atjzwlitqars27znzy1 10.255.0.5 10.255.0.6 [&PortConfig{Name:,Protocol:TCP,TargetPort:8000,PublishedPort:8000,}] [] [13ceb7ebe29e] false} err:network 3697eaa0ye8khxfed418b2z91 not found" Nov 29 17:08:21 localhost dockerd: time="2019-11-29T17:08:21.414249368+08:00" level=error msg="failed adding container name resolution for eff3bd0c83a975679a4c12c1c04f45bc7c8402e6f9c924aa1ac66cff806e66ad epRec:{ingress-endpoint <nil> 10.255.0.2 [] [] [] false} err:network 3697eaa0ye8khxfed418b2z91 not found" Nov 29 17:08:21 localhost dockerd: time="2019-11-29T17:08:21.414267965+08:00" level=warning msg="rmServiceBinding handleEpTableEvent whoami 75ef29f86a095ad6eca0e67d4f7f223586f3e41154b057e094bc49b5f2f6607d aborted c.serviceBindings[skey] !ok" Nov 29 17:08:21 localhost dockerd: time="2019-11-29T17:08:21.414287767+08:00" level=error msg="failed removing container name resolution for eff3bd0c83a975679a4c12c1c04f45bc7c8402e6f9c924aa1ac66cff806e66ad epRec:{ingress-endpoint <nil> 10.255.0.2 [] [] [] false} err:network 3697eaa0ye8khxfed418b2z91 not found"012019-11-29
相似问题