Podman storage gets corrupted when pulling an image #27035
Description
Issue Description
I'm trying to get an image to run with Podman, defined as a quadlet file.
systemctl --user start authentiek-server.service fails with exit code 125:
Sep 10 09:49:52 host authentik-server[711811]: time="2025-09-10T09:49:52+02:00" level=error msg="Image ghcr.io/goauthentik/server:2025.4.4 exists in local storage but may be corrupted (remove the image to resolve the issue): loading primary layer store data: error removing stale temp dir /home/podman/.local/share/containers/storage/overlay/tempdirs/temp-dir-1618383482: unlinkat /home/podman/.local/share/containers/storage/overlay/tempdirs/temp-dir-1618383482/1-c7ce558278b619ac541fc61a1267e58df72ccf8e624c87002b313a8a65f17207: directory not empty"
Sep 10 09:49:52 host authentik-server[711811]: Trying to pull ghcr.io/goauthentik/server:2025.4.4...
Sep 10 09:49:52 host authentik-server[711811]: Pulling image //ghcr.io/goauthentik/server:2025.4.4 inside systemd: setting pull timeout to 5m0s
Sep 10 09:49:56 host authentik-server[711811]: Getting image source signatures
Sep 10 09:49:56 host authentik-server[711811]: Copying blob sha256:15b2a750b48ae57cf27d2f075e6375860496c115f0e61bed5c7b97d3a1d54c0e
...
Sep 10 09:49:56 host authentik-server[711811]: Copying blob sha256:648fcb0e95e6aacd293cb96fc78f7feedd197e41e792f52bac248272ef46b0ce
Sep 10 09:49:56 host authentik-server[711811]: Error: unable to copy from source docker://ghcr.io/goauthentik/server:2025.4.4: copying system image from manifest list: trying to reuse blob sha256:61320b01ae5e0798393ef25f2dc72faf43703e60ba089b07d7170acbabbf8f62 at destination: looking for layers with digest "sha256:61320b01ae5e0798393ef25f2dc72faf43703e60ba089b07d7170acbabbf8f62": loading primary layer store data: error removing stale temp dir /home/podman/.local/share/containers/storage/overlay/tempdirs/temp-dir-1618383482: unlinkat /home/podman/.local/share/containers/storage/overlay/tempdirs/temp-dir-1618383482/1-c7ce558278b619ac541fc61a1267e58df72ccf8e624c87002b313a8a65f17207: directory not empty
Sep 10 09:49:56 host podman[711811]: 2025-09-10 09:49:56.132091414 +0200 CEST m=+3.537743714 image pull-error ghcr.io/goauthentik/server:2025.4.4 unable to copy from source docker://ghcr.io/goauthentik/server:2025.4.4: copying system image from manifest list: trying to reuse blob sha256:61320b01ae5e0798393ef25f2dc72faf43703e60ba089b07d7170acbabbf8f62 at destination: looking for layers with digest "sha256:61320b01ae5e0798393ef25f2dc72faf43703e60ba089b07d7170acbabbf8f62": loading primary layer store data: error removing stale temp dir /home/podman/.local/share/containers/storage/overlay/tempdirs/temp-dir-1618383482: unlinkat /home/podman/.local/share/containers/storage/overlay/tempdirs/temp-dir-1618383482/1-c7ce558278b619ac541fc61a1267e58df72ccf8e624c87002b313a8a65f17207: directory not empty
Sep 10 09:49:56 host systemd[1170]: authentik-server.service: Main process exited, code=exited, status=125/n/a
I don't think is an issue with that image in particular, I've tried a more recent one (ghcr.io/goauthentik/server:2025.8.1) and I get the same issue if I try to run ghcr.io/haveagitgat/tdarr:latest.
The issue appears only when I execute the command systemctl --user start authentiek-server.service. If I run the command podman pull ghcr.io/goauthentik/server:2025.4.4 I don't get any issue but the container fails to start with the systemctl command anyway:
podman@host:~> podman pull ghcr.io/goauthentik/server:2025.4.4
Trying to pull ghcr.io/goauthentik/server:2025.4.4...
Getting image source signatures
Copying blob 15b2a750b48a skipped: already exists
...
Copying config 96abf2164c done |
Writing manifest to image destination
96abf2164c7947bdbb3a90ca2087322fdf2979e8a29b129ada3aae08a8e9cfba
I've already tried to delete the image and pulling again, same result.
Steps to reproduce the issue
Steps to reproduce the issue
- Try to run ghcr.io/goauthentik/server:2025.4.4 or ghcr.io/haveagitgat/tdarr:latest
- Issue pulling the image when starting the container with systemctl --user command
Describe the results you received
podman@host:~> systemctl --user restart authentik-server.service
Job for authentik-server.service failed because a timeout was exceeded.
See "systemctl --user status authentik-server.service" and "journalctl --user -xeu authentik-server.service" for details.
Describe the results you expected
To have a running container from the given image
podman info output
podman@host:~> podman info
host:
arch: amd64
buildahVersion: 1.41.3
cgroupControllers:
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.13-1.2.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.13, commit: unknown'
cpuUtilization:
idlePercent: 86.75
systemPercent: 5.57
userPercent: 7.68
cpus: 4
databaseBackend: sqlite
distribution:
distribution: opensuse-tumbleweed
version: "20250903"
eventLogger: journald
freeLocks: 2004
hostname: host
idMappings:
gidmap:
- container_id: 0
host_id: 1002
size: 1
- container_id: 1
host_id: 231072
size: 65536
uidmap:
- container_id: 0
host_id: 1002
size: 1
- container_id: 1
host_id: 231072
size: 65536
kernel: 6.16.3-1-default
linkmode: dynamic
logDriver: journald
memFree: 483172352
memTotal: 7982845952
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.16.0-1.1.x86_64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.16.0
package: netavark-1.15.2-1.1.x86_64
path: /usr/libexec/podman/netavark
version: netavark 1.15.2
ociRuntime:
name: crun
package: crun-1.22-1.1.x86_64
path: /usr/bin/crun
version: |-
crun version 1.22
commit: 4de19b63a85efd9ea8503452179c371181750130
rundir: /run/user/1002/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: passt-20250611.0293c6f-3.2.x86_64
version: |
pasta 20250611.0293c6f-3.2
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: true
path: /run/user/1002/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: ""
package: ""
version: ""
swapFree: 5704228864
swapTotal: 7982841856
uptime: 11h 10m 46.00s (Approximately 0.46 days)
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.opensuse.org
- registry.suse.com
- docker.io
store:
configFile: /home/podman/.config/containers/storage.conf
containerStore:
number: 34
paused: 0
running: 34
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/podman/.local/share/containers/storage
graphRootAllocated: 255522574336
graphRootUsed: 89003753472
graphStatus:
Backing Filesystem: btrfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 31
runRoot: /run/user/1002/containers
transientStore: false
volumePath: /home/podman/.local/share/containers/storage/volumes
version:
APIVersion: 5.6.0
Built: 1756711675
BuiltTime: Mon Sep 1 09:27:55 2025
GitCommit: ""
GoVersion: go1.24.6
Os: linux
OsArch: linux/amd64
Version: 5.6.0Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
Additional environment details
Additional information
Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting