docker 1.13.1 rpc error: code = 14 the connection is unavailable 解决办法

现象

[root@slave-140 ~]# docker exec -it fe289c21a69d sh 
rpc error: code = 14 desc = grpc: the connection is unavailable
1
2

日志信息

...
Dec 11 17:03:22 slave-140 dockerd[16589]: time="2018-12-11T17:03:22.468279596+08:00" level=error msg="Create container failed with error: grpc: the connection is unavailable"

Dec 11 17:03:23 slave-140 dockerd[16589]: time="2018-12-11T17:03:23.515418042+08:00" level=error msg="Create container failed with error: grpc: the connection is unavailable"
Dec 11 17:03:23 slave-140 dockerd[16589]: time="2018-12-11T17:03:23.551705791+08:00" level=error msg="Handler for POST /v1.26/containers/78ba39f7d082e301800f3b01fd18454c9babf44c0e35a16c4c693c8044e7ef42/start returned error: grpc: the connection is unavailable"
...
1
2
3
4
5
6

怀疑docker-containerd 挂掉了，运行一下命令

[root@slave-140 ~]# docker-containerd-ctr --address unix:///run/docker/libcontainerd/docker-containerd.sock containers
[ctr] rpc error: code = 14 desc = grpc: the connection is unavailable

[root@slave-143 containerd]# docker stop 07646329caa3
Error response from daemon: Cannot stop container 07646329caa3: Cannot kill container 07646329caa3e77b64f76707df5f69242f753c678a9c344dc83ff25cdd0cdb2f: rpc error: code = 14 desc = grpc: the connection is unavailable
1
2
3
4
5

报同样的错误
删除失败

[root@slave-143 containerd]# docker rm -f 07646329caa3
Error response from daemon: Could not kill running container 07646329caa3e77b64f76707df5f69242f753c678a9c344dc83ff25cdd0cdb2f, cannot remove - Cannot kill container 07646329caa3e77b64f76707df5f69242f753c678a9c344dc83ff25cdd0cdb2f: rpc error: code = 14 desc = grpc: the connection is unavailable
1
2

最后重启解决

systemctl restart docker
1

查看

[root@slave-140 ~]# docker-containerd-ctr --address unix:///run/docker/libcontainerd/docker-containerd.sock containers
ID                                                                 PATH                                                                                             STATUS              PROCESSES
059d21f5f6f8b01e208016d1e555507bda0d2d7445121e6279d42c47a924cc22   /var/run/docker/libcontainerd/059d21f5f6f8b01e208016d1e555507bda0d2d7445121e6279d42c47a924cc22   running             init
1dceec2f287b2754447e45070b64f731dfb3348371ade45dd063debacd2a6e7a   /var/run/docker/libcontainerd/1dceec2f287b2754447e45070b64f731dfb3348371ade45dd063debacd2a6e7a   running             init
234ca5b4e2ea4418367f9078c95b933f3bfdeabb27b6472568951c40aec35415   /var/run/docker/libcontainerd/234ca5b4e2ea4418367f9078c95b933f3bfdeabb27b6472568951c40aec35415   running             init
26331cd78ac87231a70c74352e853d6f954c3a351b096f1ac4e6dedd36bcca3b   /var/run/docker/libcontainerd/26331cd78ac87231a70c74352e853d6f954c3a351b096f1ac4e6dedd36bcca3b   running             init
26fe102fd6b3ae0b880920e4ba27004445f149b36b37b322ee524ecd5f12d345   /var/run/docker/libcontainerd/26fe102fd6b3ae0b880920e4ba27004445f149b36b37b322ee524ecd5f12d345   running             init
27bedf8fc9bce7733221f7f93e6780195b84231b53b8a54bc0e489ad12832b15   /var/run/docker/libcontainerd/27bedf8fc9bce7733221f7f93e6780195b84231b53b8a54bc0e489ad12832b15   running             init
280fb94624c2e357e95cb267f128ac4f3628636b2feb8f5
...
1
2
3
4
5
6
7
8
9
10

正常

相关issue Can’t run new containers - grpc: the connection is unavailable.

container not stop, rpc error: code = 14 desc = grpc: the connection is unavailable, v1.13.1

[17.03.x] Update containerd to 595e75c212d19a81d2b808a518fe1afc1391dad5