Volume list order inconsistencies between swarm commands forces service restarts #49673
Description
Description
In Docker Swarm, if a service is deployed with volumes that are not listed alphabetically in the compose file, executing docker service update on the service causes it to restart and appears to re-order the volumes alphabetically. If docker stack deploy is then executed, it will again restart the service and list the volumes in the original order.
I have tested and observed this behavior on both v25.x.x and v28.0.1.
Reproduce
Create a vol1 and vol2 directory in the same path as the following compose.yaml. Note the volumes are not listed alphabetically.
.
├── compose.yaml
├── vol1
└── vol2
compose.yaml:
version: '3.8'
services:
alpine:
image: alpine:latest
command: 'tail -f /dev/null'
deploy:
replicas: 1
volumes:
- type: bind
source: ./vol2
target: /mnt/vol2
- type: bind
source: ./vol1
target: /mnt/vol1Create a swarm with the test stack:
docker swarm init
docker stack deploy -c compose.yaml test_stackNote that the volumes are still listed out of order:
# docker service inspect test_stack_alpine | jq '.[0].Spec.TaskTemplate.ContainerSpec.Mounts'
[
{
"Type": "bind",
"Source": "/host_mnt/username/swarm/vol2",
"Target": "/mnt/vol2"
}
{
"Type": "bind",
"Source": "/host_mnt/username/swarm/vol1",
"Target": "/mnt/vol1"
},
]Now update the service - the service will restart and the volumes will now be listed alphabetically:
# Terminal 1
watch docker service ps test_stack_alpine
# Terminal 2
docker service update test_stack_alpine
docker service inspect test_stack_alpine | jq '.[0].Spec.TaskTemplate.ContainerSpec.Mounts' # Notice mount order has changedIf you re-deploy the stack, the container will again restart and the mount order will go back to what it was orginally:
docker stack deploy -c compose.yaml test_stack
docker service inspect test_stack_alpine | jq '.[0].Spec.TaskTemplate.ContainerSpec.Mounts'Expected behavior
Services should not be restarted by docker stack deploy nor docker service update if the volume configuration has not changed.
docker version
Client: Docker Engine - Community
Version: 25.0.3
API version: 1.44
Go version: go1.21.6
Git commit: 4debf41
Built: Tue Feb 6 21:13:09 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 25.0.3
API version: 1.44 (minimum version 1.24)
Go version: go1.21.6
Git commit: f417435
Built: Tue Feb 6 21:13:09 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.25
GitCommit: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc:
Version: 1.2.4
GitCommit: v1.2.4-0-g6c52b3f
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client: Docker Engine - Community
Version: 25.0.3
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.20.0
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.6
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 54
Running: 34
Paused: 0
Stopped: 20
Images: 68
Server Version: 25.0.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: local
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: active
NodeID: 29h0vzw6lxdosc4zkotn5qa9s
Is Manager: false
Node Address: 10.10.2.190
Manager Addresses:
10.10.2.235:2377
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc version: v1.2.4-0-g6c52b3f
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-1021-aws
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 48
Total Memory: 185.7GiB
Name: ip-10-10-2-190
ID: 63079822-15c0-4a89-a717-bf93b5d310ad
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: <redacted>
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Default Address Pools:
Base: 192.168.0.0/17, Size: 25Additional Info
This bug seems similar to #32344, except with volumes instead of environment variables.
As a workaround, defining the volumes in your compose file in the order that docker service update lists them seems to prevent services from being restarted.
Metadata
Metadata
Assignees
Type
Projects
Status