containerd: "docker pull" from private registry with S3 compatible backend fails #50065
Description
Description
I'm trying to pull an image on MacOS from our private Gitlab registry with a S3 compatible Minio backend. Several HTTP Requests succeeds but one fails with status code 400.
When I disable containderd image storage, then pulling works perfectly.
I used tcpdump to trace the HTTP requests on the Minio server. A couple of successful HTTP requests look like this:
GET /gitlab-images/docker/registry/v2/blobs/sha256/3c/3ccc92c3be6175b1c12713b0c125b0cc49aba8d7852b6cdf93d336770775381d/data HTTP/1.1
Host: images.bic.myhost.tld
User-Agent: docker-distribution/v4.21.0-gitlab (go1.23.9) aws-sdk-go/1.55.5 (go1.23.9; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXX/20250524/us-east-1/s3/aws4_request, SignedHeaders=host;range;x-amz-content-sha256;x-amz-date, Signature=XXXXXXThen this one fails with status code 400 because it contains two authentication types:
GET /gitlab-images/docker/registry/v2/blobs/sha256/f1/f14ee681fe6418a8a69642b3a741e4a50c9b569f2dce4235f9344f8737467bf3/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXXXXX%2F20250524%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20250524T085244Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=XXXXXXX HTTP/1.1
Host: images.bic.mydomain.tld
User-Agent: docker/28.1.1 go/go1.23.8 git-commit/01f442b kernel/6.10.14-linuxkit os/linux arch/arm64 containerd-client/2.0.5+unknown storage-driver/overlayfs UpstreamClient(Docker-Client/28.1.1 \(darwin\))
Accept: application/vnd.docker.container.image.v1+json, */*
Accept-Encoding: zstd;q=1.0, gzip;q=0.8, deflate;q=0.5
Authorization: Bearer [2600 bytes]
Baggage: trigger=api<Error>
<Code>InvalidRequest</Code>
<Message>Invalid Request (request has multiple authentication types, please use one)</Message>
<Resource>/gitlab-images/docker/registry/v2/blobs/sha256/f1/f14ee681fe6418a8a69642b3a741e4a50c9b569f2dce4235f9344f8737467bf3/data</Resource>
<RequestId></RequestId>
<HostId>6041f9a1-841a-49f9-ad08-299d080713eb</HostId>
</Error>Reproduce
- docker login bic.myhost.tld
- docker pull bic.myhost.tld/group/project:latest
Expected behavior
Pulling should succeed.
docker version
Client:
Version: 28.1.1
API version: 1.49
Go version: go1.23.8
Git commit: 4eba377
Built: Fri Apr 18 09:49:45 2025
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.41.2 (191736)
Engine:
Version: 28.1.1
API version: 1.49 (minimum version 1.24)
Go version: go1.23.8
Git commit: 01f442b
Built: Fri Apr 18 09:52:08 2025
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.27
GitCommit: 05044ec0a9a75232cad458027ca83437aae3f4da
runc:
Version: 1.2.5
GitCommit: v1.2.5-0-g59923ef
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client:
Version: 28.1.1
Context: desktop-linux
Debug Mode: false
Plugins:
ai: Docker AI Agent - Ask Gordon (Docker Inc.)
Version: v1.1.7
Path: /Users/julian/.docker/cli-plugins/docker-ai
buildx: Docker Buildx (Docker Inc.)
Version: v0.23.0-desktop.1
Path: /Users/julian/.docker/cli-plugins/docker-buildx
cloud: Docker Cloud (Docker Inc.)
Version: v0.3.0
Path: /Users/julian/.docker/cli-plugins/docker-cloud
compose: Docker Compose (Docker Inc.)
Version: v2.36.0-desktop.1
Path: /Users/julian/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.38
Path: /Users/julian/.docker/cli-plugins/docker-debug
desktop: Docker Desktop commands (Docker Inc.)
Version: v0.1.8
Path: /Users/julian/.docker/cli-plugins/docker-desktop
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: /Users/julian/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.27
Path: /Users/julian/.docker/cli-plugins/docker-extension
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.4.0
Path: /Users/julian/.docker/cli-plugins/docker-init
mcp: Docker MCP Plugin (Docker Inc.)
Version: dev
Path: /Users/julian/.docker/cli-plugins/docker-mcp
model: Docker Model Runner (Docker Inc.)
Version: v0.1.23
Path: /Users/julian/.docker/cli-plugins/docker-model
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/julian/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.18.0
Path: /Users/julian/.docker/cli-plugins/docker-scout
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 28.1.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
runc version: v1.2.5-0-g59923ef
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.10.14-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 12
Total Memory: 7.653GiB
Name: docker-desktop
ID: 60bf5177-33ef-4888-bedf-b484738c221a
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=unix:///Users/julian/Library/Containers/com.docker.docker/Data/docker-cli.sock
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
::1/128
127.0.0.0/8
Live Restore Enabled: false
WARNING: DOCKER_INSECURE_NO_IPTABLES_RAW is set
WARNING: daemon is not using the default seccomp profileAdditional Info
I've reported this issue in the docker/cli repository before:
docker/cli#6106