Problems with "Smart remove" in "Run in background on remote Host. EXPERIMENTAL!" when to much snapshots to delete #1359
Description
Thanks to @aryoda comment in PR #1351 I found that issue.
BIT has an "EXPERIMENTAL" feature where the "smart remove" is done in background via SSH.
Usually it isn't done in background. And based on my observations it isn't done with SSH even if an SSH snapshot profiles is used. The remote is mounted and then a simple rm (more precise shutil.rmtree()) is done. No SSH is involed.
The problem is that the feature seems really to be experimental. In my test scenario it creates a command more than 10.000 characters long which seems to be to long for SSH. Here you see just a snipped example.
['ssh', '-o', 'ServerAliveInterval=240', '-o', 'LogLevel=Error', '-o', 'IdentityFile=/home/user/.ssh/id_rsa', '-p', '22', 'user@localhost', 'screen -d -m bash -c "(TMP=\\$(mktemp -d); test -z \\"\\$TMP\\" && exit 1; test -n \\"\\$(ls \\$TMP)\\" && exit 1; logger -t \\"backintime smart-remove [$BASHPID]\\" \\"start\\"; flock -x 9; logger -t \\"backintime smart-remove [$BASHPID]\\" \\"got exclusive flock\\"; test -e \\"/home/user/dest blank/backintime/localhost/user/1/20221106-214507-663\\" && (logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221106-214507-663 still exist\\"; sleep 1; rsync -a --delete -s \\"\\$TMP/\\" /home/user/dest blank/backintime/localhost/user/1/20221106-214507-663; rmdir \\"/home/user/dest blank/backintime/localhost/user/1/20221106-214507-663\\"; logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221106-214507-663 remove done\\"); test -e \\"/home/user/dest blank/backintime/localhost/user/1/20221105-214500-483\\" && (logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221105-214500-483 still exist\\"; sleep 1; rsync -a --delete -s \\"\\$TMP/\\" /home/user/dest blank/backintime/localhost/user/1/20221105-214500-483; rmdir \\"/home/user/dest blank/backintime/localhost/user/1/20221105-214500-483\\"; logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221105-214500-483 remove done\\"); test -e \\"/home/user/dest blank/backintime/localhost/user/1/20221104-221056-108\\" && (logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221104-221056-108 still exist\\"; sleep 1; rsync -a --delete -s \\"\\$TMP/\\"
SNIPPED
&& (logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221019-230601-797 still exist\\"; sleep 1; rsync -a --delete -s \\"\\$TMP/\\" /home/user/dest blank/backintime/localhost/user/1/20221019-230601-797; rmdir \\"/home/user/dest blank/backintime/localhost/user/1/20221019-230601-797\\"; logger -t \\"backintime smart-remove [$BASHPID]\\" \\"snapshot 20221019-230601-797 remove done\\"); rmdir \\$TMP) 9>\\"/home/user/dest blank/backintime/localhost/user/1/smartremove.lck\\""']
The problem is obvious and also some solutions are possible. But when this problem is solved maybe there are other problems coming up with the experimental feature. I suspect that this will take much more time and testing resources.
My first suggestion was to "deactivate" that feature for the 1.3.3 release. And after that release we have more time to dive into it.
Maybe @Germar can give us more background info about that feature.
A problem could be. That the feature was introduced with 1.1.6 in the year 2015 ([CHANGELOG](* add option to run Smart Remove in background on remote host (https://launchpad.net/bugs/1457210)) and #257). so it is 7 years old and could be more stable than experimental.