Files within /sbin are occasionally empty after an upgrade #1758
Description
Describe the bug
Running into an occasional issue where the /sbin/ directory appears to have all of its contents, but the files have a size of 0 bytes.
Originally we thought this might be an issue with our custom init script, as the kernel was panicking when it couldn't access /sbin/preinit.sh. Further investigation showed that this was the case for most files in /sbin as well, this script just happens to be the first we attempt to access.
Background information
# rauc --version
rauc 1.13
We are building for Yocto 5.0 (scarthgap) on an i.MX8M Plus platform. Linux kernel version is 6.6.23.
To Reproduce
Steps to reproduce the behavior:
- Install an upgrade bundle (
rauc install <filename>.raucb) - Check contents of /sbin on the newly updated slot
- Easiest to do this with a post-install hook since it keeps the target slot mounted.
test -s ${RAUC_SLOT_MOUNT_POINT}/sbin/<target filename>- returns 1 if file does not exist or is 0 bytes
- Repeat steps 1 and 2 until the issue occurs.
Expected behavior
The files in /sbin are non-empty.
Logs
If applicable, add console output or logs from the RAUC service to help explain your problem.
ext4ls on a failed slot
Verdin iMX8MP Mainboard# ext4ls mmc 2:1 /sbin
<DIR> 4096 .
<DIR> 4096 ..
0 agetty
0 badblocks
0 cgdisk
0 debugfs
<SYM> 8 dosfsck
<SYM> 8 dosfslabel
<SYM> 15 dropbear
<SYM> 15 dropbearconvert
<SYM> 15 dropbearkey
0 dropbearmulti
0 dumpe2fs
0 e2freefrag
0 e2fsck
0 e2image
0 e2mmpstatus
0 e2undo
0 e4crypt
0 e4defrag
0 eeprog
0 ethtool
0 faillock
0 fatlabel
0 filefrag
0 fixparts
0 fsck.fat
<SYM> 8 fsck.msdos
0 fsck.util-linux
<SYM> 8 fsck.vfat
0 gdisk
0 hdparm.hdparm
0 i2cdetect.i2c-tools
0 i2cdump.i2c-tools
0 i2cget.i2c-tools
0 i2cset.i2c-tools
0 i2ctransfer.i2c-tools
0 ldconfig
0 logrotate
0 logsave
<SYM> 8 mkdosfs
0 mke2fs.e2fsprogs
0 mkfs.fat
<SYM> 8 mkfs.msdos
<SYM> 8 mkfs.vfat.dosfstools
0 mkhomedir_helper
0 mklost+found
0 mount-copybind
0 pam_namespace_helper
0 pam_timestamp_check
0 populate-extfs.sh
0 preinit.sh
0 pwhistory_helper
0 rsyslogd
0 sgdisk
0 sulogin.util-linux
0 swapoff.util-linux
0 swapon.util-linux
0 sysctl.procps
0 unix_chkpwd
0 unix_update
0 update-ca-certificates
<SYM> 23 addgroup
<SYM> 23 adduser
<SYM> 23 blkid
<SYM> 23 chroot
<SYM> 23 delgroup
<SYM> 23 deluser
<SYM> 21 depmod
<SYM> 11 depmod.kmod
<SYM> 23 fbset
<SYM> 23 fdisk
<SYM> 25 fsck
0 fsck.ext2
0 fsck.ext3
0 fsck.ext4
<SYM> 23 fstrim
...
ext4ls on a known-good slot
Verdin iMX8MP Mainboard# ext4ls mmc 2:2 /sbin
<DIR> 4096 .
<DIR> 4096 ..
134392 agetty
67576 badblocks
264344 cgdisk
271864 debugfs
<SYM> 8 dosfsck
<SYM> 8 dosfslabel
<SYM> 15 dropbear
<SYM> 15 dropbearconvert
<SYM> 15 dropbearkey
397016 dropbearmulti
67568 dumpe2fs
67560 e2freefrag
413616 e2fsck
67760 e2image
67568 e2mmpstatus
67552 e2undo
67632 e4crypt
67560 e4defrag
67552 eeprog
724464 ethtool
67560 faillock
67640 fatlabel
67584 filefrag
67736 fixparts
133176 fsck.fat
<SYM> 8 fsck.msdos
67672 fsck.util-linux
<SYM> 8 fsck.vfat
264344 gdisk
135304 hdparm.hdparm
67640 i2cdetect.i2c-tools
67640 i2cdump.i2c-tools
67640 i2cget.i2c-tools
67640 i2cset.i2c-tools
67648 i2ctransfer.i2c-tools
795096 ldconfig
133184 logrotate
67560 logsave
<SYM> 8 mkdosfs
198944 mke2fs.e2fsprogs
68168 mkfs.fat
<SYM> 8 mkfs.msdos
<SYM> 8 mkfs.vfat.dosfstools
75776 mkhomedir_helper
67560 mklost+found
2387 mount-copybind
467 pam_namespace_helper
67568 pam_timestamp_check
2460 populate-extfs.sh
673 preinit.sh
67568 pwhistory_helper
817504 rsyslogd
264344 sgdisk
67648 sulogin.util-linux
67640 swapoff.util-linux
67648 swapon.util-linux
67688 sysctl.procps
67560 unix_chkpwd
67560 unix_update
6039 update-ca-certificates
<SYM> 23 addgroup
<SYM> 23 adduser
<SYM> 23 blkid
<SYM> 23 chroot
<SYM> 23 delgroup
<SYM> 23 deluser
<SYM> 21 depmod
<SYM> 11 depmod.kmod
<SYM> 23 fbset
<SYM> 23 fdisk
<SYM> 25 fsck
413616 fsck.ext2
413616 fsck.ext3
413616 fsck.ext4
<SYM> 23 fstrim
...
Additional context
Problem was exacerbated at first due to a bug in our U-Boot config which prevented slot boot attempts from being decremented (which has since been fixed).
We have implemented a post-install hook that checks for this issue, and exits with an error code if the init script does not exist or is 0 bytes. Unfortunately realized afterwards that this will not change the success/failure status of the install: https://rauc.readthedocs.io/en/stable/using.html#system-based-customization-handlers