Nothing Special   »   [go: up one dir, main page]
Skip to content
This repository has been archived by the owner on Mar 9, 2022. It is now read-only.

failed to create containerd task #1082

Closed
keyingliu opened this issue Mar 11, 2019 · 7 comments
Closed

failed to create containerd task #1082

keyingliu opened this issue Mar 11, 2019 · 7 comments
Labels
kind/bug
Milestone
v1.0

Comments

@keyingliu
Copy link
keyingliu commented Mar 11, 2019 •
edited
Loading

Some of the containers cannot create with following error, the containerd we are running included PR #1072:

          starting container process caused "process_linux.go:424: container init
          caused \"rootfs_linux.go:58: mounting \\\"/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/549442c80aed91a09d82b7f8274d36290dca4a80993bf2ba4af0801344f8c8e3/hostname\\\"
          to rootfs \\\"/run/containerd/io.containerd.runtime.v1.linux/k8s.io/7b6f62d1e8f31fc297777f45616c7e566275d40a1398ea3556bcd173d56a8806/rootfs\\\"
          at \\\"/run/containerd/io.containerd.runtime.v1.linux/k8s.io/7b6f62d1e8f31fc297777f45616c7e566275d40a1398ea3556bcd173d56a8806/rootfs/etc/hostname\\\"
          caused \\\"not a directory\\\"\"": unknown

I can see the ${container_id}/hostname is a directory in the host, but it should be a file.
@keyingliu
Copy link
Author

checked that the ${sandbox_id}/hostname is created during sandbox startup, but need to figure out why this file is missing...

@Random-Liu
Copy link
Member

@keyingliu This shouldn't happen. We have an integration test testing this behavior https://github.com/containerd/cri/blob/master/integration/pod_hostname_test.go#L89.

What image are you using? It could be related to your image.

@keyingliu
Copy link
Author

@Random-Liu I just picked one of our nodes, in /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes, run ls -R:

./1dfcec412f1752ca253b0208496e713861386e216798459ddd92265a1e170ab9:
hostname  hosts  resolv.conf

./358a9876da034f741649be1aa8262b65fd22bd1d67d63e77c77ec2c5039fe13d:
hosts  resolv.conf

./6c67c91e2a66c3d52ad80a001a32b642c7e8d121b98843028476e3e0f94e4c4b:
hostname  hosts  resolv.conf

./8224d8cba753739f6cc525154ac927b017b5a42d4cc22e9e498413928582ec8b:
hosts  resolv.conf

./90b4739bb3f11aa74ada0b1c2adbf93c0ae5265f1ddbdf6c2a43ca391b2144bd:
hostname  hosts  resolv.conf

./90b4739bb3f11aa74ada0b1c2adbf93c0ae5265f1ddbdf6c2a43ca391b2144bd/hostname:

./9182a2bf5d919bb520557b83f1963a4e552fb1933907bed281f3d230c2acbf79:
hosts  resolv.conf

./9407d3f3bc0099a77469e839dc044a06e0b2e8569d9def6513c10b1a884e1b6f:
hosts  resolv.conf

./a71b13e5fc3be7651a99e6ae688e094c12550073804699c9bbc4bb140b9dda9c:
hosts  resolv.conf

./aa1b8912f8a9052ccf0e5812d9f059f976c21d9cfea52cb9a37ca9bb8879405c:
hosts  resolv.conf

./ae42a6bf74489c8ad1643097827a0ec92ef9924b0adbd2a2ee9dffe9510e54f1:
hosts  resolv.conf

./af2b06fb2c3b18582ff8c730065bc12275311c2472c056b81936b4eb877ceef0:
hosts  resolv.conf

./ba55e2c222ceedcef812fd44ea685577912cfa32271792753a654997efb524f7:
hosts  resolv.conf

./cd54b904d55c36b6b1835fbd09a37cd5e8193d0ec6b0c7afc358c10c5cff9a4b:
hosts  resolv.conf

./d29cf2f181b0c3c836e78ddcd4b3bf4bb99a1c07a755ada4dabab01001f53b40:
hostname  hosts  resolv.conf

./d4cd2a46f8db6ee5c0a9bd545eba543be0d10dc3c95c3237a2bb214f94a648f4:
hostname  hosts  resolv.conf

./ee4239cbb7592d56344be7720a4ff4aabbe58be759f33742e3a74e42558c13b6:
hosts  resolv.conf

./f2052d65e620a9c5115571c4e8e97e4beb7f487d6ac5a02aef29f7d037df7f85:
hosts  resolv.conf

Take care the sandbox: 90b4739bb3f11aa74ada0b1c2adbf93c0ae5265f1ddbdf6c2a43ca391b2144bd, the hostname is a folder. and just wondering why some of the sandboxes have file hostname, but some of them missing the file.

@Random-Liu
Copy link
Member
Random-Liu commented Mar 12, 2019 •
edited
Loading

@keyingliu I see... Did you update containerd with all existing services running?

The hostname change is not a backward compatible change... If you have an old sandbox running, there won't be hostname file.

In this case, it is fine to not have hostname in container, but at least container should be able to start.

BTW, for production environment, it is always recommended to drain the node before updating critical node components. :)

@Random-Liu Random-Liu added this to the v1.0 milestone Mar 12, 2019
@Random-Liu Random-Liu mentioned this issue Mar 12, 2019
Merged
@Random-Liu
Copy link
Member

@keyingliu See #1085. Will include it in 1.2.5.

@keyingliu
Copy link
Author

@Random-Liu Thanks for root cause analysis and fix.

@chengyli
Copy link

@Random-Liu When will containerd 1.2.5 be released?

@Random-Liu Random-Liu mentioned this issue Apr 2, 2019
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
v1.0
Development

No branches or pull requests
3 participants
@Random-Liu @keyingliu @chengyli