Nothing Special   »   [go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Containerd Image pull fails intermittently with EOF error for Harbor private registry #8971

Closed
charugarg2810 opened this issue Aug 16, 2023 · 23 comments
Closed

Containerd Image pull fails intermittently with EOF error for Harbor private registry #8971

charugarg2810 opened this issue Aug 16, 2023 · 23 comments
Assignees
@fuweid
Labels
area/cri Container Runtime Interface (CRI) kind/bug Stale status/more-info-needed Awaiting contributor information

Comments

@charugarg2810
Copy link
charugarg2810 commented Aug 16, 2023 •
edited
Loading

Description

Hi Team,

We have recently upgraded our EKS cluster from v1.23 to 1.24, which replaces the Docker with Containerd. Now in 1.24 clusters, we are seeing imagepullbackoff issues for harbor images. While trying to pull the image manually it gives below output.

Steps to reproduce the issue

Describe the results you received and expected

ctr -n k8s.io image pull harbor.registry.dns/image/path:1234 --user user1

Password:
harbor.registry.dns: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:371b1e63285f74ff6927becf34a47f302e954c7835987c0660c63b8ac5bb4651: waiting |--------------------------------------|
manifest-sha256:1ec1ffc82b90a3e9cb7445c2079cb60a954b41303b0ac9174144ded2f4cccc05: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:d437a0ad48fceb1463e1c2af0b35959ab5cbab3a4961009d1e964571d7d9e161: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f868ac632a6f7994c589ce631afb99014698a783e173144bf4740e318303e992: downloading |++++++++++++++++++++++++++++++++------| 214.0 Mi/250.5 MiB
layer-sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0b172db2480d88fcc4bec429da2777b56687dbcd82d80c19dc41e16ffc56d14b: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:76bf1f85fdf38a127d6047c9b3d96651797e2cb5825172dee23cb611f3111f9c: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fe906bbc5faed6202ead1ec035b9467f3cf02bf5120145dec708061c5e81c119: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:39182aedab0a7dad5dbd70262fbc13e54f2b2cab4f278078650936c2e7dae52a: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:381187300573beeff328abb31687c357112e575c9ca02b256683f98224cc7e93: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:52c3e4aa8e82ec89a0cedcea37e8d9b4541b00d560d85874cb271fb80288e75b: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e25e5e21d0a6a269d82a220ce0848c2862694dbee4956386a06f5e71ad8f892c: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:16c640aa63795ffac2c65bdb68e5d81cbf3b640862f710151f1b0a8e4d570753: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.4 s total: 230.5 (96.0 MiB/s)
ctr: failed to read expected number of bytes: unexpected EOF

What version of containerd are you using?

1.6.19

Any other relevant information

runc --version

runc version 1.1.5
commit: f19387a6bec4944c770f7668ab51c4348d9c2f38
spec: 1.0.2-dev
go: go1.18.6
libseccomp: 2.4.1

uname -a
5.10.179-171.711.amzn2.aarch64 #1 SMP Tue Jun 6 01:59:12 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Show configuration if it is related to CRI plugin.

No response
@fuweid
Copy link
Member
fuweid commented Aug 16, 2023

ctr: failed to read expected number of bytes: unexpected EOF

I think it might be related to your registry. Would you mind to use debug mode to rerun it again? thanks

@fuweid fuweid added the status/more-info-needed Awaiting contributor information label Aug 16, 2023
@charugarg2810
Copy link
Author

# ctr --debug -n k8s.io image pull $image --user user1 harbor.registry.dns:latest sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f Password: DEBU[0001] fetching image="harbor.registry.dns/project1:198-307d1e4" DEBU[0001] resolving host=harbor.registry.dns DEBU[0001] do request host=harbor.registry.dnsrequest.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.19 request.method=HEAD url="https://harbor.registry.dns/v2/project1/manifests/198-307d1e4" DEBU[0001] fetch response received host=harbor.registry.dns response.header.connection=keep-alive response.header.content-length=743 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Tue, 25 Jul 2023 12:34:47 GMT" response.header.docker-content-digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=01ebd46f-8fd7-488b-9e63-4dcdd3933d95 response.status="200 OK" url="https://harbor.registry.dns/v2/project1/manifests/198-307d1e4" DEBU[0001] resolved desc.digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" host=harbor.registry.dns DEBU[0001] fetch digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=743 DEBU[0001] do request digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.accept="application/vnd.docker.distribution.manifest.list.v2+json, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=743 url="https://harbor.registry.dns/v2/project1/manifests/sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" DEBU[0001] fetch response received digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.connection=keep-alive response.header.content-length=743 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Tue, 25 Jul 2023 12:34:47 GMT" response.header.docker-content-digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=213557ed-5882-4eeb-861c-9f4c3dee00f1 response.status="200 OK" size=743 url="https://harbor.registry.dns/v2/project1/manifests/sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" DEBU[0001] fetch digest="sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" mediatype=application/vnd.docker.distribution.manifest.v2+json size=4510 DEBU[0001] do request digest="sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=4510 url="https://harbor.registry.dns/v2/project1/manifests/sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" DEBU[0001] fetch response received digest="sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.connection=keep-alive response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.vary=Accept-Encoding response.header.x-request-id=9e10eb41-ed01-44fa-b6e9-6c2ed402ef32 response.status="200 OK" size=4510 url="https://harbor.registry.dns/v2/project1/manifests/sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" DEBU[0001] fetch digest="sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=13666138 DEBU[0001] fetch digest="sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=161807505 DEBU[0001] fetch digest="sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=260 DEBU[0001] fetch digest="sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=5400162 DEBU[0001] fetch digest="sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=4194546 DEBU[0001] fetch digest="sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=5869988 DEBU[0001] fetch digest="sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1413 DEBU[0001] fetch digest="sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=40917255 DEBU[0001] fetch digest="sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" mediatype=application/vnd.docker.container.image.v1+json size=8424 DEBU[0001] fetch digest="sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11517 DEBU[0001] fetch digest="sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=921635 DEBU[0001] fetch digest="sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=156125588 DEBU[0001] fetch digest="sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=5108 DEBU[0001] fetch digest="sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=6551 DEBU[0001] fetch digest="sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=4561395 DEBU[0001] fetch digest="sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1001 DEBU[0001] fetch digest="sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=980 DEBU[0001] fetch digest="sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=292 DEBU[0001] fetch digest="sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=32 DEBU[0001] fetch digest="sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=5118 DEBU[0001] fetch digest="sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=148608462 DEBU[0001] do request digest="sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=13666138 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" DEBU[0001] do request digest="sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=161807505 url="https://harbor.registry.dns/v2/project1/blobs/sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" DEBU[0001] do request digest="sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=260 url="https://harbor.registry.dns/v2/project1/blobs/sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" DEBU[0001] fetch response received digest="sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=13666138 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=930504fd-e1ed-46e1-9fce-ce5edf55f2cf response.status="200 OK" size=13666138 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" DEBU[0001] do request digest="sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=4194546 url="https://harbor.registry.dns/v2/project1/blobs/sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" DEBU[0001] do request digest="sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=5400162 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" DEBU[0001] do request digest="sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=921635 url="https://harbor.registry.dns/v2/project1/blobs/sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" DEBU[0001] do request digest="sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=5869988 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" DEBU[0001] do request digest="sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=1413 url="https://harbor.registry.dns/v2/project1/blobs/sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" DEBU[0001] fetch response received digest="sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=260 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=2830f2e7-6952-4a29-864c-c7a89eb8c41f response.status="200 OK" size=260 url="https://harbor.registry.dns/v2/project1/blobs/sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" DEBU[0001] do request digest="sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=156125588 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" DEBU[0001] do request digest="sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=40917255 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" DEBU[0001] fetch response received digest="sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=161807505 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=fb8f6a07-74c8-4118-856a-72fcf7b5ec9c response.status="200 OK" size=161807505 url="https://harbor.registry.dns/v2/project1/blobs/sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" DEBU[0001] fetch response received digest="sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=921635 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=a7014d4e-7a52-4d06-8b35-c2de02edede8 response.status="200 OK"size=921635 url="https://harbor.registry.dns/v2/project1/blobs/sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" DEBU[0001] fetch response received digest="sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=4194546 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=5f60a1dc-2250-4b9c-b0cb-0065276429e1 response.status="200 OK" size=4194546 url="https://harbor.registry.dns/v2/project1/blobs/sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" DEBU[0001] do request digest="sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=5108 url="https://harbor.registry.dns/v2/project1/blobs/sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" DEBU[0001] do request digest="sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=8424 url="https://harbor.registry.dns/v2/project1/blobs/sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" DEBU[0001] fetch response received digest="sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=5400162 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=cafddd3e-edef-4a28-955a-b74703c6ac27 response.status="200 OK" size=5400162 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" DEBU[0001] fetch response received digest="sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=5869988 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=990996c1-085f-4458-b04e-47cdd9f49e02 response.status="200 OK" size=5869988 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" DEBU[0001] fetch response received digest="sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=1413 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=5d030e7d-e717-4c1d-b172-c89ca1ab74db response.status="200 OK" size=1413 url="https://harbor.registry.dns/v2/project1/blobs/sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" DEBU[0001] fetch response received digest="sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=40917255 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=409986c6-43c6-4984-9454-c9114a894f6a response.status="200 OK" size=40917255 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" DEBU[0001] do request digest="sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=6551 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" DEBU[0001] do request digest="sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=11517 url="https://harbor.registry.dns/v2/project1/blobs/sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" DEBU[0001] fetch response received digest="sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=5108 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=d3d8c186-f8d5-4dbd-bce6-ba1d8a7beebe response.status="200 OK" size=5108 url="https://harbor.registry.dns/v2/project1/blobs/sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" DEBU[0001] fetch response received digest="sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=156125588 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=0a4414e4-0523-4cd4-ad9b-3a0ce18201b8 response.status="200 OK" size=156125588 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" DEBU[0001] fetch response received digest="sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=8424 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=fdf14afd-0326-4e8d-89a1-915a1ed69f14 response.status="200 OK" size=8424 url="https://harbor.registry.dns/v2/project1/blobs/sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" DEBU[0001] do request digest="sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=4561395 url="https://harbor.registry.dns/v2/project1/blobs/sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" DEBU[0001] fetch response received digest="sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=6551 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=cd5e2e65-2781-4e51-9e94-90aacb0e5864 response.status="200 OK" size=6551 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" DEBU[0001] do request digest="sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=980 url="https://harbor.registry.dns/v2/project1/blobs/sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" DEBU[0001] fetch response received digest="sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=4561395 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=1c8db2f3-2f79-4d7b-ac30-584be6a69486 response.status="200 OK" size=4561395 url="https://harbor.registry.dns/v2/project1/blobs/sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" DEBU[0001] do request digest="sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=292 url="https://harbor.registry.dns/v2/project1/blobs/sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" DEBU[0001] fetch response received digest="sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=11517 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=bc41631d-96c9-4c45-8dd9-c5d5a3d3fcb7 response.status="200 OK" size=11517 url="https://harbor.registry.dns/v2/project1/blobs/sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" DEBU[0001] fetch response received digest="sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=980 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=df7a8699-24d1-4207-9bdb-ba32fae5e3d9 response.status="200 OK" size=980 url="https://harbor.registry.dns/v2/project1/blobs/sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" DEBU[0001] fetch response received digest="sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=292 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=c1ffacca-e94b-4e05-be04-6bbb16ce4237 response.status="200 OK" size=292 url="https://harbor.registry.dns/v2/project1/blobs/sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" DEBU[0001] do request digest="sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=1001 url="https://harbor.registry.dns/v2/project1/blobs/sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" DEBU[0001] do request digest="sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=148608462 url="https://harbor.registry.dns/v2/project1/blobs/sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" DEBU[0001] fetch response received digest="sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=1001 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=41d4d7d9-3af5-40f8-ac52-a983f54149df response.status="200 OK" size=1001 url="https://harbor.registry.dns/v2/project1/blobs/sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" DEBU[0001] fetch response received digest="sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=148608462 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=6ec16589-6746-4e8e-af86-bf2fca7aea7e response.status="200 OK" size=148608462 url="https://harbor.registry.dns/v2/project1/blobs/sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" DEBU[0001] do request digest="sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=5118 url="https://harbor.registry.dns/v2/project1/blobs/sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" DEBU[0001] fetch response received digest="sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=5118 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=6c3bc69e-38e7-4d58-8de0-ec1a3072586b response.status="200 OK" size=5118 url="https://harbor.registry.dns/v2/project1/blobs/sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" DEBU[0001] do request digest="sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=2097152-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=156125588 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" ctr: failed to read expected number of bytes: unexpected EOF

@fuweid
Copy link
Member
fuweid commented Aug 18, 2023

layer-sha256:f868ac632a6f7994c589ce631afb99014698a783e173144bf4740e318303e992: downloading |++++++++++++++++++++++++++++++++------| 214.0 Mi/250.5 MiB

@charugarg2810 I think there is some wrong with this blob. Would you mind to use ctr -n k8s.io leases delete leaky lease first and retry it? thanks

@charugarg2810
Copy link
Author

The thing is it's an intermittent issue and is happening for multiple images. On some retries (~ 10-15 times) it gets auto-resolved.

@charugarg2810
Copy link
Author

Below is the debug output for some different image:
ctr --debug -n k8s.io image pull $image --user user1 Password: DEBU[0001] fetching image="harbor.registry.dns/project2:3684-cfafb9c28" DEBU[0001] resolving host=harbor.registry.dns DEBU[0001] do request host=harbor.registry.dns request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.19 request.method=HEAD url="https://harbor.registry.dns/project1/manifests/3684-cfafb9c28" DEBU[0001] fetch response received host=harbor.registry.dns response.header.connection=keep-alive response.header.content-length=192 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 20 Jul 2023 07:12:13 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx/1.16.1 response.header.set-cookie="sid=aa9dcd5312a404b7e8bfc8dc4f155a2e; Path=/; HttpOnly; Secure" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.www-authenticate="Basic realm=\"harbor\"" response.header.x-request-id=a795fab4-841c-4567-a84f-206c7f2daaca response.status="401 Unauthorized" url="https://harbor.registry.dns/project1/manifests/3684-cfafb9c28" DEBU[0001] Unauthorized header="Basic realm=\"harbor\"" host=harbor.registry.dns DEBU[0001] do request host=harbor.registry.dns request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.19 request.method=HEAD url="https://harbor.registry.dns/project1/manifests/3684-cfafb9c28" DEBU[0001] fetch response received host=harbor.registry.dns response.header.connection=keep-alive response.header.content-length=743 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Thu, 20 Jul 2023 07:12:14 GMT" response.header.docker-content-digest="sha256:df0f0bca49e367088b4985fcc32d5cb0abf5a15db1c531d6a82c4b06d9eb4994" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:df0f0bca49e367088b4985fcc32d5cb0abf5a15db1c531d6a82c4b06d9eb4994\"" response.header.server=nginx/1.16.1 response.header.set-cookie="sid=e217064197c1ad5ab47e3b4a2adca8bb; Path=/; HttpOnly; Secure" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=b835fd09-a879-4248-9fb5-492fbbaad07a response.status="200 OK" url="https://harbor.registry.dns/project1/manifests/3684-cfafb9c28" DEBU[0001] resolved desc.digest="sha256:df0f0bca49e367088b4985fcc32d5cb0abf5a15db1c531d6a82c4b06d9eb4994" host=harbor.registry.dns DEBU[0001] fetch digest="sha256:df0f0bca49e367088b4985fcc32d5cb0abf5a15db1c531d6a82c4b06d9eb4994" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=743 DEBU[0001] fetch digest="sha256:5dc741397828b0a3c5d40b843a39a6fe78a3fc08c52f98fdc75d829161338fa7" mediatype=application/vnd.docker.distribution.manifest.v2+json size=5562 DEBU[0001] fetch digest="sha256:5d0f5cb39866238d754e5aa8277ee9654c7663c18928d053021e5b67235c3f4f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=159 DEBU[0001] fetch digest="sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=204219065 DEBU[0001] fetch digest="sha256:0862d060a21322221192c0217ea66bf9f43f6b9ebbba874cc6fa5aeb3d64bc31" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=5460683 DEBU[0001] fetch digest="sha256:0c2892dfbf98327e72cce129ac1201e5df5b14b97bb36e753e5a0b121d31b293" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=13686681 DEBU[0001] fetch digest="sha256:e66dd0b135f39400f503752b1f1f8b27743eac7db2efa92cf76ff54acdb0083c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1274731 DEBU[0001] fetch digest="sha256:03688cf8216857c77f07905f4a93f1dd5145f32a3d9bc1084719aa4b6e7f6dce" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=4268383 DEBU[0001] fetch digest="sha256:9d361db69dbdf93606f541b86c7d04dd72c4f390b6c73712b21732daa2016095" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=2419117 DEBU[0001] fetch digest="sha256:e8aae28818c31cd4cde3387c229fbab0255b21bdf14fabe61681eda6510be1ce" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=921644 DEBU[0001] fetch digest="sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=193390970 DEBU[0001] fetch digest="sha256:4ddda290423598192b382b88f44cc18d2c76c05f20ffff871424024f58e1791c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=260 DEBU[0001] fetch digest="sha256:24ead560a6c43a1632281366a49c39bcd11a25ce37575e8dc98e885ccff53cce" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=263511 DEBU[0001] fetch digest="sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=148608514 DEBU[0001] fetch digest="sha256:64a523d3310c30b60c073e29ede5429ce136cd2448cf50a2cec5f1f931b5a1e9" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1085 DEBU[0001] fetch digest="sha256:964cd7fa8adf4b7670619524a55ff185687c274cdb3ff778059bb82ee994e9a3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=6551 DEBU[0001] fetch digest="sha256:868cd957f1d366a1193d16fcf3f3920fa93fd020e5b19ea82b8099214456c6f6" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=6922 DEBU[0001] fetch digest="sha256:e5c8d896e5d455eab4b94c49a82d6a3837b44363e702b4128ed0a5f88e260dcf" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=290 DEBU[0001] fetch digest="sha256:d9c9f3854d13b863781575605521d72fff338e2cce6b73986448a4313af1e625" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=4818075 DEBU[0001] fetch digest="sha256:070da07b1bf535bdfc2e7f9b70669d58cb61e1744c437917ea911bc007767105" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1348 DEBU[0001] fetch digest="sha256:fa539d310f66603be77222b7950e77f164d7ddc102b6d64434483c997144af18" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=12326034 DEBU[0001] fetch digest="sha256:c58ba5199dfd2e9135e3f9381c56b8f7068475b982155c42fbdd31b6b72e8873" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=6928 DEBU[0001] fetch digest="sha256:7b3aba51fd01a060fdd122a2317d78de0463d9ffbf97c267051b09411d6e859c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=40917426 DEBU[0001] fetch digest="sha256:9ee2c1ac3140e49bea6912b70b2bdfffe08e0450d89d885115abff5f5b9ec3d6" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=4561393 DEBU[0001] fetch digest="sha256:06556d2c8f3fc9205db5f6c8ffaddb6811cb8c378ef452b03c2ca0c97673b858" mediatype=application/vnd.docker.container.image.v1+json size=9948 DEBU[0001] fetch digest="sha256:56f3752ba3fc19b9baf5228ee50dc8b23853f5f02ed41135ac08dadc095da587" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=6711223 DEBU[0001] fetch digest="sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=32 DEBU[0001] fetch digest="sha256:782a561fc530865ea2df30e7fcf23ad30e318c3038609bafbbf6b3fd53ec4e8d" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=93 DEBU[0001] do request digest="sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=6291456-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=204219065 url="https://harbor.registry.dns/project1/blobs/sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" DEBU[0001] do request digest="sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=6291456-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=193390970 url="https://harbor.registry.dns/project1/blobs/sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" DEBU[0002] do request digest="sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=31457280-" request.header.user-agent=containerd/1.6.19 request.method=GET size=148608514 url="https://harbor.registry.dns/project1/blobs/sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" DEBU[0002] do request digest="sha256:fa539d310f66603be77222b7950e77f164d7ddc102b6d64434483c997144af18" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=6291456-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=12326034 url="https://harbor.registry.dns/project1/blobs/sha256:fa539d310f66603be77222b7950e77f164d7ddc102b6d64434483c997144af18" DEBU[0002] fetch response received digest="sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=117151234 response.header.content-range="bytes 31457280-148608513/148608514" response.header.content-type=application/octet-stream response.header.date="Thu, 20 Jul 2023 07:12:14 GMT" response.header.docker-content-digest="sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=47250330-6e4d-4e32-897b-22e76bfe435e response.status="206 Partial Content" size=148608514 url="https://harbor.registry.dns/project1/blobs/sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" DEBU[0002] do request digest="sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=33554432-" request.header.user-agent=containerd/1.6.19 request.method=GET size=148608514 url="https://harbor.registry.dns/project1/blobs/sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" DEBU[0003] fetch response received digest="sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.connection=keep-alive response.header.content-length=192 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 20 Jul 2023 07:12:15 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx/1.16.1 response.header.set-cookie="sid=11c5295a976eca3a9624aedfcd02c8b5; Path=/; HttpOnly; Secure" response.header.set-cookie.1="sid=e989ad272a2829e6141415413b8bec71; Path=/; HttpOnly; Secure" response.header.set-cookie.2="sid=3adfc5eb029ab1f05c8eaad6b69a82a3; Path=/; HttpOnly;Secure" response.header.set-cookie.3="sid=d9e8834c3ebc4e9359541f568ef6cc27; Path=/; HttpOnly; Secure" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.www-authenticate="Basic realm=\"harbor\"" response.header.x-request-id=c077a6c7-9dbb-4d1c-9ba2-debfb9abe41b response.status="401 Unauthorized" size=204219065 url="https://harbor.registry.dns/project1/blobs/sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" DEBU[0003] Unauthorized digest="sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" header="Basic realm=\"harbor\"" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=204219065 DEBU[0003] do request digest="sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=6291456-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=204219065 url="https://harbor.registry.dns/project1/blobs/sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" DEBU[0003] fetch response received digest="sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.connection=keep-alive response.header.content-length=192 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 20 Jul 2023 07:12:15 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx/1.16.1 response.header.set-cookie="sid=a35e264d6dbe8d99f072770c3e538b16; Path=/; HttpOnly; Secure" response.header.set-cookie.1="sid=c88d40c0bda300f38c30cda15e69627e; Path=/; HttpOnly; Secure" response.header.set-cookie.2="sid=a318f94af285e9a880c461a27b8ef271; Path=/; HttpOnly;Secure" response.header.set-cookie.3="sid=81cd99e2cd6a6b39f4bafae5dc36b283; Path=/; HttpOnly; Secure" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.www-authenticate="Basic realm=\"harbor\"" response.header.x-request-id=38def421-0688-48ca-977b-364be7fae2ca response.status="401 Unauthorized" size=193390970 url="https://harbor.registry.dns/project1/blobs/sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" DEBU[0003] Unauthorized digest="sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" header="Basic realm=\"harbor\"" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=193390970 DEBU[0003] do request digest="sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=6291456-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=193390970 url="https://harbor.registry.dns/project1/blobs/sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" ctr: failed to read expected number of bytes: unexpected EOF

@fuweid
Copy link
Member
fuweid commented Aug 18, 2023

On some retries (~ 10-15 times) it gets auto-resolved.

So, I think it maybe related to your registry. :(

@charugarg2810
Copy link
Author

The only concern is it's working fine with Docker. Only Conatinerd is failing.

@djdongjin
Copy link
Member

Maybe try containerd v1.6.23 https://github.com/containerd/containerd/tree/v1.6.23 and see if the issue still exists.

Context: we got EOF errors before (not sure if it's the same root cause), and made a fix #8800 which should be included in 1.6.23.

@charugarg2810
Copy link
Author

I tried but with no success.

root@ip-172-21-33-56 ~]# /usr/local/bin/ctr -n k8s.io image pull $image --user usr1
Password:
harbor.registry.dns/project1:3699-a809bb499:                resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:d6a6ad06523b574f25306250670c90ac5316ecb6d0978867a783b7750b1f7ff1:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:922f3e2012b77f830697a58603fc2ec6e9f3ea5e4605dba61b480ebdf5bd6601: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:03688cf8216857c77f07905f4a93f1dd5145f32a3d9bc1084719aa4b6e7f6dce:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:444fa77c7578f20995274254c12a615ab06041ffbb155242d19944978185ea59:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0f6a07685401b9afe3b0791681ad041b69f8021c941c174b4520a629101bf7b3:    downloading    |+-------------------------------------|  6.0 MiB/194.8 MiB
layer-sha256:7b3aba51fd01a060fdd122a2317d78de0463d9ffbf97c267051b09411d6e859c:    downloading    |+++++++++++++++++++++-----------------| 22.0 MiB/39.0 MiB
layer-sha256:0c2892dfbf98327e72cce129ac1201e5df5b14b97bb36e753e5a0b121d31b293:    downloading    |+++++++++++++++++---------------------|  6.0 MiB/13.1 MiB
layer-sha256:eeda2982daa94b080aa882e79942ca697594994a32302ab316e8f8fd80048592:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9c7ded1cc7e593c8a488f25d5272993352c1f9724a184cf4730ecf332d9ab635:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e8aae28818c31cd4cde3387c229fbab0255b21bdf14fabe61681eda6510be1ce:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3315ac82074d0ef4e31a0c74a82d68e07c28f41d750836fc4806496b0cae1d14:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:13c22b252c64e1120ce74d4e091e29940e8480532f4d55971f1cab5757b8e862:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4ddda290423598192b382b88f44cc18d2c76c05f20ffff871424024f58e1791c:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3e7ec20fc0634a378976a8edc0e82354446819925dc0e5a287c19e2601aa823e:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:12d7196581569a9b12b1fd774cc20c3dc07154c1d3145d324749144db3ba053c:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea:    downloading    |--------------------------------------|  2.0 MiB/141.7 MiB
layer-sha256:e0d2f29fcf852715b0534d30ae5ef14b4dd774947b479bf36486f08e6635e492:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1:    waiting        |--------------------------------------|
layer-sha256:c8a5cd881d8ae893c2ed54fc96bc2f0703c9271056488acce23ad317d930bd48:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:964cd7fa8adf4b7670619524a55ff185687c274cdb3ff778059bb82ee994e9a3:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b86064b015f08be41226d7a586f039711a357b498a5d040f83d144becf902020:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:44d2315972efb3f4d353b27393441271c3a3f7b56531fb3e4af5c3274e256732:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bbbdedd4d4581a125f50348660142fed0990444377c0136392023a26b6c243ee:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bba9d6338fff27ce8138e8085d4c09e5816f72fc1b2c2f0163800a33e9eca774:    downloading    |+-------------------------------------|  6.0 MiB/184.4 MiB
layer-sha256:e5c8d896e5d455eab4b94c49a82d6a3837b44363e702b4128ed0a5f88e260dcf:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:618e0d39adef9038f7437a861ca250f529291fd03af0c3e58597f1a844f06ae9:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9ee2c1ac3140e49bea6912b70b2bdfffe08e0450d89d885115abff5f5b9ec3d6:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:dff02559314f94cc184ecd2858a9a7435faf549a8b2c50789e371b90b3b1b63e:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.6 s                                                                    total:  83.1 M (137.9 MiB/s)
ctr: failed to read expected number of bytes: unexpected EOF
[root@ip-172-21-33-56 ~]#
[root@ip-172-21-33-56 ~]#
[root@ip-172-21-33-56 ~]# ctr --version
ctr github.com/containerd/containerd v1.6.23
[root@ip-172-21-33-56 ~]#

@d4n13lbc
Copy link

Hi. I have the same issue with EKS 1.25 and containerd version 1.6.19

@charugarg2810
Copy link
Author

@d4n13lbc you too using the Harbor Docker registry?

@starpipi
Copy link
starpipi commented Sep 9, 2023

@charugarg2810
I use Harbor Docker registry too.
I can pull images with ctr or nerdctl,But crictl can't pull. Just shown EOF
I tried containerd version 1.7.5 and 1.6.23. It's the same.

@srinathrangaramanujam
Copy link

We have the same issue eith eks 1.25.

@kmursk
Copy link
kmursk commented Oct 13, 2023

Fatal error... New versions of kubernetes use crictl only, but crictl is not able to pull images by https. While this issue is not resolved, the single solution i had found is to switch harbor to work through http and force containerd do the same.

@rbjorklin
Copy link

We are seeing the same intermittent issue with JFrog Artifactory.

@rbjorklin
Copy link
rbjorklin commented Nov 21, 2023 •
edited
Loading

Might this be a regression in containerd? In migrating our container runtime we made a rather significant version bump.
We used to be on Docker 20.10.11 which bundled containerd=1.4.12 and jumped all the way to containerd=1.6.20.

What's the situation for everyone else who have chimed in on this issue?

EDIT: For what it's worth it looks like there is an integration test checking for this error.

@xiang90
Copy link
xiang90 commented Jan 4, 2024

Hit the same issue with containerd 1.7.11 with AWS ECR

See

  Warning  Failed            58m                 kubelet            Failed to pull image "xxx": rpc error: code = Unavailable desc = error reading from server: EOF

@fuweid fuweid self-assigned this Jan 4, 2024
@fuweid fuweid added the area/cri Container Runtime Interface (CRI) label Jan 4, 2024
@github-actions GitHub Actions
Copy link
github-actions bot commented Apr 4, 2024

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 7 days unless new comments are made or the stale label is removed.

@github-actions github-actions bot added the Stale label Apr 4, 2024
@rbjorklin
Copy link

Not stale.

@github-actions github-actions bot removed the Stale label Apr 5, 2024
@yulinor
Copy link
yulinor commented Jun 25, 2024

The same error is reported, distribution v2.8.3 is used as a cache pull warehouse, and the node uses containerd 1.6.23.
Occasionally there will be an image that cannot be pulled on all nodes, but it can be pulled after restarting the distribution.

I strongly suspect that there is a problem with distribution, but the info/debug logs of the service are all Get HTTP requests, filesystem.getcontent,filesystem.Reader, filesystem.putcontent, response completed, filesystem.Delete repeatedly,no error log output

@artemijan
Copy link

I noticed the same issue in our build system, the root cause was that we messed up with docker cache and loaded (docker load) wrong images to build main app.

@github-actions GitHub Actions
Copy link

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 7 days unless new comments are made or the stale label is removed.

@github-actions github-actions bot added the Stale label Nov 13, 2024
@github-actions GitHub Actions
Copy link

This issue was closed because it has been stalled for 7 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fuweid fuweid

Labels
area/cri Container Runtime Interface (CRI) kind/bug Stale status/more-info-needed Awaiting contributor information
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests