Nothing Special   »   [go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Containerd panics with fatal error: concurrent map writes when pull images concurrently #8742

Closed
shuaichang opened this issue Jun 27, 2023 · 0 comments · Fixed by #8748
Closed

Containerd panics with fatal error: concurrent map writes when pull images concurrently #8742

shuaichang opened this issue Jun 27, 2023 · 0 comments · Fixed by #8748
Labels
kind/bug

Comments

@shuaichang
Copy link
Contributor
shuaichang commented Jun 27, 2023 •
edited
Loading

Description

Containerd paniced when during concurrent image pulling. I believe the following is the root cause, but feel free to correct me if I missed anything:

Since c.config.Registry.Headers is a global map, concurrent image pull could result in concurrent map.Set

containerd/pkg/cri/server/image_pull.go

Line 133 in 9b4ed8a

Headers: c.config.Registry.Headers,

options.Headers.Set("User-Agent", "containerd/"+version.Version) could result to concurrent map write
https://github.com/containerd/containerd/blob/9b4ed8acc2a04a3f8df140e79052d18b750d757e/remotes/docker/resolver.go#L153C1-L153C1

Steps to reproduce the issue

We had a client using CRI client calling PullImage almost at the same time and can get a repro.

Describe the results you received and expected

Got a panic when pulling images concurrently when calling CRI.PullImage API. Expecting pullImage to be thread safe

Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: fatal error: concurrent map writes
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: goroutine 339 [running]:
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: runtime.throw({0x55c6b3076599?, 0x10?})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /usr/local/go/src/runtime/panic.go:992 +0x71 fp=0xc000174c08 sp=0xc000174bd8 pc=0x55c6b1df5e91
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: runtime.mapassign_faststr(0x55c6b306211c?, 0x55c6b306404b?, {0x55c6b306211c, 0xa})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /usr/local/go/src/runtime/map_faststr.go:212 +0x39c fp=0xc000174c70 sp=0xc000174c08 pc=0x55c6b1dd21bc
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: net/textproto.MIMEHeader.Set(...)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /usr/local/go/src/net/textproto/header.go:22
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: net/http.Header.Set(...)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /usr/local/go/src/net/http/header.go:40
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/containerd/containerd/remotes/docker.NewResolver({0xc0006b08b8, 0xc000038750, {0x7efc08032300, 0xc0006b08d0}, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/remotes/docker/resolver.go:145 +0x225 fp=0xc000174e48 sp=0xc000174c70 pc=0x55c6b25724a5
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/containerd/containerd/pkg/cri/server.(*criService).PullImage(0xc000601800, {0x55c6b3902d70, 0xc00055a0c0}, 0xc00082c180)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/pkg/cri/server/image_pull.go:104 +0x2a5 fp=0xc000175228 sp=0xc000174e48 pc=0x55c6b2f17825
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/containerd/containerd/pkg/cri/server.(*instrumentedAlphaService).PullImage(0xc00011a7b0, {0x55c6b3902d70, 0xc0005b3e90}, 0xc000634f00)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/pkg/cri/server/instrumented_service.go:966 +0x277 fp=0xc000175480 sp=0xc000175228 pc=0x55c6b2f2d1f7
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: k8s.io/cri-api/pkg/apis/runtime/v1alpha2._ImageService_PullImage_Handler.func1({0x55c6b3902d70, 0xc0005b3e90}, {0x55c6b3871400?, 0xc000634f00})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go:9855 +0x78 fp=0xc0001754c0 sp=0xc000175480 pc=0x55c6b2de24d8
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/containerd/containerd/services/server.unaryNamespaceInterceptor({0x55c6b3902d70, 0xc0005b3e90}, {0x55c6b3871400, 0xc000634f00}, 0x3?, 0xc0006b07e0)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/services/server/namespace.go:31 +0x6b fp=0xc0001754f0 sp=0xc0001754c0 pc=0x55c6b3012dcb
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x55c6b3902d70?, 0xc0005b3e90?}, {0x55c6b3871400?, 0xc000634f00?})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0x3a fp=0xc000175530 sp=0xc0001754f0 pc=0x55c6b3009a9a
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1({0x55c6b3902d70, 0xc0005b3e90}, {0x55c6b3871400, 0xc000634f00}, 0x0?, 0xc00082c020)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0x87 fp=0xc000175590 sp=0xc000175530 pc=0x55c6b300c5e7
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x55c6b3902d70?, 0xc0005b3e90?}, {0x55c6b3871400?, 0xc000634f00?})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0x3a fp=0xc0001755d0 sp=0xc000175590 pc=0x55c6b3009a9a
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x55c6b3902d70, 0xc0005b3c20}, {0x55c6b3871400, 0xc000634f00}, 0xc00082c000, 0xc00082c040)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:325 +0x664 fp=0xc000175a10 sp=0xc0001755d0 pc=0x55c6b3010204
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x55c6b3902d70?, 0xc0005b3c20?}, {0x55c6b3871400?, 0xc000634f00?})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0x3a fp=0xc000175a50 sp=0xc000175a10 pc=0x55c6b3009a9a
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x55c6b3902d70, 0xc0005b3c20}, {0x55c6b3871400, 0xc000634f00}, 0xc000470af0?, 0x55c6b36b3620?)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0xbf fp=0xc000175aa8 sp=0xc000175a50 pc=0x55c6b300993f
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: k8s.io/cri-api/pkg/apis/runtime/v1alpha2._ImageService_PullImage_Handler({0x55c6b38a8f40?, 0xc00011a7b0}, {0x55c6b3902d70, 0xc0005b3c20}, 0xc000516480, 0xc0002e41b0)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go:9857 +0x138 fp=0xc000175b00 sp=0xc000175aa8 pc=0x55c6b2de2398
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000140a80, {0x55c6b3908ba0, 0xc0006c3520}, 0xc000638120, 0xc000158600, 0x55c6b44e71f0, 0x0)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1283 +0xcfd fp=0xc000175e48 sp=0xc000175b00 pc=0x55c6b2458e5d
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: google.golang.org/grpc.(*Server).handleStream(0xc000140a80, {0x55c6b3908ba0, 0xc0006c3520}, 0xc000638120, 0x0)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1620 +0xa1b fp=0xc000175f68 sp=0xc000175e48 pc=0x55c6b245d4bb
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: google.golang.org/grpc.(*Server).serveStreams.func1.2()
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:922 +0x98 fp=0xc000175fe0 sp=0xc000175f68 pc=0x55c6b2456978
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: runtime.goexit()
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000175fe8 sp=0xc000175fe0 pc=0x55c6b1e2abc1
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:920 +0x28a
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: goroutine 1 [chan receive]:
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/containerd/containerd/cmd/containerd/command.App.func1(0xc00016e2c0?)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/cmd/containerd/command/main.go:279 +0x134c
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/urfave/cli.HandleAction({0x55c6b361e2e0?, 0x55c6b38dfac8?}, 0xc00003c230?)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:523 +0xa8
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: github.com/urfave/cli.(*App).Run(0xc000140700, {0xc00003c230, 0x1, 0x1})
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:285 +0x818
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: main.main()
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         github.com/containerd/containerd/cmd/containerd/main.go:33 +0x37
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: goroutine 11 [chan receive]:
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x0?)
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]: created by k8s.io/klog/v2.init.0
Jun 26 22:02:47 ip-10-7-161-36 containerd[1691]:         /go/src/github.com/containerd/containerd/vendor/k8s.io/klog/v2/klog.go:420 +0xf6

What version of containerd are you using?

1.6.9.4

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Resolve docker.NewResolver race condition djdongjin/containerd
1 participant
@shuaichang