chore(backup): add s3 backup script #818

max-wittig · 2017-05-05T08:17:59Z

max-wittig · 2017-05-05T10:26:09Z

Successfully tested on dev-fossology

maxhbr · 2017-07-12T07:55:24Z

utils/backup/S3/fo-backup-s3

+fi
+
+if [[ ${1} = "-l" ]]; then
+ LOGGING=${2}


It might be better to call this variable LOG or LOGFILE

maxhbr · 2017-07-12T07:58:06Z

Please do not create wiki pages before the corresponding features are merged.

maxhbr

some comments

maxhbr · 2017-07-19T11:41:36Z

utils/backup/S3/fo-backup-s3

+
+echo Start database dump at `date`
+#### postgresql needs to be restarted for the dump to work
+/etc/init.d/postgresql restart


why not just /etc/init.d/postgresql start.

because it needs to be restarted for the dump to work. I don't think start would work

I would not know why this should be necessary (except for reloading the config, which you do not change)

or maybe this forces the DB to write everything out...

maxhbr · 2017-07-19T11:46:09Z

utils/backup/S3/fo-backup-s3

+
+#### sync repository
+echo "Uploading and starting backup to S3..."
+tar -zcvf - /srv/fossology/ | ${DIR}/venv/bin/aws --debug s3 cp --expected-size 200000000000 --sse AES256 - "s3://${BUCKET}/${FILENAME}"


--sse AES256 is the default value

Yes, but maybe the default value will change overtime and then it wouldn't work anymore. What do you think?

maxhbr · 2017-07-19T12:09:32Z

utils/backup/S3/fo-backup-s3

+
+
+echo Creating diff of repository at `date`
+ls -RlahGg --time-style='+' /srv/fossology/repository/ | cut -d ' ' -f 2-5 | ${DIR}/venv/bin/aws s3 cp --sse AES256 - "s3://${BUCKET}/${FILENAME}-repo-diff.txt"


Why is this recursive? This cut splits folders with spaces in their names in arbitrary places.
Why do you keep the inode-count?

Please add variables for magic strings like "s3://${BUCKET}/${FILENAME}-repo-diff.txt" which are used in multiple places.

splits folders with spaces in their names in arbitrary places

No it just splits the column. There is a tab in there, not spaces.

Since you are listing recursively, the ls -RlahGg --time-style='+' also lists the names of the corresponding folders as the first "word" in a line (for sub directories). If they contain spaces you are splitting them and keeping a part of them.

Do you happen to know a way to remove the inode-count, besides cut?

example:

$ tree ./ /tmp/test ./ ├── Folder with splaces │ └── file └── otherFile

and

$ ls -RlahGg --time-style='+' . | cut -d ' ' -f 2-5 /tmp/test .: 20K 3 4,0K 16 12K 2 4,0K 1 with splaces: 8,0K 2 4,0K . 3 4,0K .. 1

Note:

here you also see that the ls -RlahGg --time-style='+' contains on my system spaces and not tabs => the filenames are not listed.

you're right. Also we could also just use tree for the diff, right? Only filesize would be gone then. Or what do you think should be the best way to approach this?

If you want a list of all files together with their sizes use something like

find /srv/fossology/repository/ -type f -exec du -h {} \;

maxhbr · 2017-07-19T12:10:22Z

utils/backup/S3/fo-restore-s3

+
+echo Checking diff of repository at `date`
+DIFF=`${DIR}/venv/bin/aws s3 cp --sse AES256 s3://${BUCKET}/${FILENAME}-repo-diff.txt -`
+CURRENT_DIFF=`ls -RlahGg --time-style='+' /srv/fossology/repository/ | cut -d ' ' -f 2-5`


You are calling ls -RlahGg --time-style='+' /srv/fossology/repository/ at multiple places, please move them to an function.

maxhbr · 2017-07-19T12:11:47Z

utils/backup/S3/fo-restore-s3

+
+echo Start removing old data at `date`
+
+${DIR}/../../fo-cleanold --delete-repository --delete-database


You might need to ensure that the database is running.
This might delete productive data, you should display a warning and give the option to exit.

maxhbr · 2017-07-19T12:13:37Z

utils/backup/S3/fo-backup-s3

+# 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+source ${DIR}/before


You are sourcing the before before setting set -x. You can't be sure that $FILENAME is filled and that apache is stopped

maxhbr · 2017-07-19T12:15:15Z

utils/backup/S3/fo-restore-s3

+  su postgres -c "psql -f ${TMP_FILE}"
+  echo --------- End database log ---------
+else
+  su postgres -c "psql -f ${TMP_FILE}" > /dev/null 2>&1


why do you also suppress STDERR?

because there are many errors that come up, because tables already exist etc...

maxhbr · 2017-07-19T12:21:41Z

utils/backup/S3/fo-backup-s3

+#### postgresql needs to be restarted for the dump to work
+/etc/init.d/postgresql restart
+#### these steps are split, because of permissions
+TMP_FILE=`mktemp`


already done in before

maxhbr · 2017-09-01T15:00:16Z

utils/backup/S3/fo-restore-s3

+  echo "Please specify a file, where to save the log file to"
+  exit 1
+ fi
+ else


indentation is confusing here

maxhbr · 2017-09-01T15:01:42Z

utils/backup/S3/fo-restore-s3

+  exit 1
+ fi
+ else
+  exec &> >(tee -a ${LOGGING})


you only call this command if no argument "-l" was passed. That looks wrong

same in the other file

mcjaeger · 2017-09-12T15:55:02Z

@max-wittig how about writing the status of this pull request as of now?

max-wittig · 2017-09-13T07:18:00Z

The Pull Request is working, but it's not possible to run the restore and backup script at the same time.
There is no safety measure to ensure that a full backup is downloaded.

max-wittig force-pushed the dev/s3-backup branch 2 times, most recently from 8ff6953 to 01448f7 Compare May 5, 2017 10:17

max-wittig changed the title ~~WIP: chore(backup): add s3 backup script~~ chore(backup): add s3 backup script May 5, 2017

max-wittig force-pushed the dev/s3-backup branch 6 times, most recently from b976e33 to 7245350 Compare May 12, 2017 09:37

mcjaeger assigned max-wittig May 16, 2017

max-wittig force-pushed the dev/s3-backup branch 7 times, most recently from a4fbdd8 to 46143d5 Compare June 20, 2017 12:08

max-wittig force-pushed the dev/s3-backup branch 12 times, most recently from c8e9cd2 to 275bdbe Compare July 6, 2017 13:19

maxhbr reviewed Jul 12, 2017

View reviewed changes

max-wittig force-pushed the dev/s3-backup branch 6 times, most recently from 2cfa4c3 to 7e0d014 Compare July 13, 2017 14:36

maxhbr added the do not merge - нет! label Jul 17, 2017

max-wittig force-pushed the dev/s3-backup branch 2 times, most recently from 8edf22a to cc4e2be Compare July 18, 2017 14:54

maxhbr reviewed Jul 19, 2017

View reviewed changes

max-wittig force-pushed the dev/s3-backup branch 4 times, most recently from 034a6ff to ff3bb1a Compare July 26, 2017 06:24

maxhbr added the WIP label Jul 27, 2017

max-wittig force-pushed the dev/s3-backup branch from 0e75722 to 20f476f Compare August 7, 2017 11:54

max-wittig force-pushed the dev/s3-backup branch 4 times, most recently from fc88e17 to e08f5bd Compare August 31, 2017 08:07

maxhbr reviewed Sep 1, 2017

View reviewed changes

max-wittig force-pushed the dev/s3-backup branch from e08f5bd to cc7e6d9 Compare September 13, 2017 09:40

feat(backup): add s3 backup and restore

4716837

max-wittig force-pushed the dev/s3-backup branch from cc7e6d9 to 4716837 Compare September 27, 2017 11:36

mcjaeger merged commit 48b72a4 into fossology:master Sep 28, 2017



		echo Creating diff of repository at `date`
		ls -RlahGg --time-style='+' /srv/fossology/repository/ \| cut -d ' ' -f 2-5 \| ${DIR}/venv/bin/aws s3 cp --sse AES256 - "s3://${BUCKET}/${FILENAME}-repo-diff.txt"


		echo Start removing old data at `date`

		${DIR}/../../fo-cleanold --delete-repository --delete-database

chore(backup): add s3 backup script #818

chore(backup): add s3 backup script #818

Uh oh!

Conversation

max-wittig commented May 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

max-wittig commented May 5, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxhbr commented Jul 12, 2017

Uh oh!

maxhbr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxhbr Jul 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

max-wittig Jul 19, 2017 • edited by maxhbr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxhbr Jul 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

example:

Note:

Uh oh!

max-wittig Jul 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxhbr Jul 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mcjaeger commented Sep 12, 2017

Uh oh!

max-wittig commented Sep 13, 2017

Uh oh!

Reviewers

Assignees

max-wittig commented May 5, 2017 •

edited

Loading

maxhbr Jul 19, 2017 •

edited

Loading

max-wittig Jul 19, 2017 •

edited by maxhbr

Loading

maxhbr Jul 19, 2017 •

edited

Loading

max-wittig Jul 19, 2017 •

edited

Loading

maxhbr Jul 19, 2017 •

edited

Loading