Lets make a killer backup tool

Postby **viking60** » 23 Jan 2011, 20:10

I am a fairly simple Magazine reader so I never understood all those fancy backup solutions out there. Mandriva had some crappy daemons and drakbackup does fill up your HD to the point where Linux gets unresponsive.

So I had a look at Bacula - Yeah right :shock:

I went for a Whiskey after 10 minutes and dropped it.
(It seems to be a great tool so it is me being unfair and impatient here).
And even If I understood how to operate different backup programs I never fully understood how they operated. To the point that I did not know where to look for files for restoring.
And I see all those competent experts out there talking about backups and I get the impression they do it every day with ease and total control. That leaves sloppy me!

Well actually I have found a way that I can control: I copy my home directory to an external USB Harddisk (My Book) and I know it works because I have restored it when necessary.
So that works - kind of. And should disaster strike I can always reinstall the distro and All the programs and create my User (puh..) restore the home directory and all is back.

It can be done a lot easier than that you say? Yes it can, and I will not be knowing what I am doing - and I hate that. :-x

But back to the simple aproach and this is where you gurus can be really useful:

I want this thread to end with a script or program that we have made here (OK I can do all the scripting but I prefer not to).
So what do we need to back up to make a sensible restore when disaster strikes? (Not all of it! Reinstalling the distro is fine).
What media to use? Can we use dropbox
How can we make use of cron jobs?
Network solutions -ssh?
Anything else?

SInce Berserks are brutal and undemocratic I'll start and dictate that we must make use of rsync. I have not made a script I just write this line every time:

Code: Select all

rsync --delete -azvv -e /home/thomas /media/My?Book/thomas-pc_backup

This backs up my entire home directory (thomas) to the attached My Book in a folder I have named thomas-pc_backup.
It works fine I can put back whatever I want from it. This is useful when I have destroyed Virtual box installations etc.

When I have My Book attached to to my server I make use of ssh:

Code: Select all

rsync --delete -azvv -e ssh /home/thomas thomas@viking60-server:/media/My?Book/thomas-pc_backup

And start it from the same box. The only differense is that the My Book is attached to the server.
That works fine to. Can we think of a solution with dropbox here?

As you can see I need to automate this stuff in a script and I thought I'll add some fancy stuff that you have helped me with.
What other areas should be backed up? Lets start there.
The stage is yours

PS if you want to go directly to the succesfull conclusion (pt) you can go here
Or you can download it from github
You will find the wiki here

Postby **rolf** » 23 Jan 2011, 22:54

I'll just throw a few strands of what I do against the wall, see if any of it sticks. Maybe you can make use of some bits.

1TB Seagate SATA drive in an external Icy Dock enclosure, USB2 or Firewire connectors; I'm using Firewire.
Mount point, /backup, that I make owned by my normal user, so not to have to be root to do backups.
Icy Dock is always connected, can be turned on/off with a power switch. When I'm ready to do a backup, I watch the following output in konsole as I turn on the switch:
Code: Select all
$ tailf /var/log/messages Jan 23 13:41:30 localhost kernel: scsi7 : SBP-2 IEEE-1394 Jan 23 13:41:31 localhost kernel: ieee1394: sbp2: Logged into SBP-2 device Jan 23 13:41:31 localhost kernel: ieee1394: sbp2: Node 0-00:1023: Max speed [S400] - Max payload [2048] Jan 23 13:41:31 localhost kernel: scsi 7:0:0:0: Direct-Access ST310005 28AS CC35 PQ: 0 ANSI: 4 Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: Attached scsi generic sg8 type 0 Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Write Protect is off Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Cache data unavailable Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Assuming drive cache: write through Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Cache data unavailable Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Assuming drive cache: write through Jan 23 13:41:31 localhost kernel: sdg: sdg1 sdg2 < sdg5 > Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Cache data unavailable Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Assuming drive cache: write through Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Attached SCSI disk

and, knowing the appropriate partition, I will become root in order to mount it, something like:
Code: Select all
# mount /dev/sdg1 /backup
Finally, a little bash script I massaged into life, with Google's help, many moons ago.
Code: Select all
for i in `cat rolf.backup`; do rsync -avz --delete --exclude 'default/kykd3xj8.slt/Cache' --exclude 'seamonkey/w1o08opq.default/Cache' --exclude 'lan/handycam' $i /backup/rolf/; done

where /rolf is a directory I created at the root of the partition on the Seagate.

My copy of `man rsync` has always had a different syntax for --exclude and it hasn't ever worked for me. Faced with 24G of lan's handycam now on disk in the path of my backup scheme, I googled what is there, now, and it seems correct. This is crude, a work in progress, but good enough for my current purposes. Here, in the same directory as I issue the bash command, I keep a text file with the paths I want to archive, rolf.backup
Code: Select all
[rolf@localhost rsync]$ cat rolf.backup /home/rolf/.gftp /home/rolf/taxes /home/rolf/.mozilla /home/rolf/addresses /home/rolf/documents /home/rolf/envelope /home/rolf/humor /home/rolf/info /home/rolf/jobs /home/rolf/maps /home/rolf/movie /home/rolf/mp3 /home/rolf/paul /home/rolf/personal /home/rolf/pictures /home/rolf/pmfax /home/rolf/rpm /home/rolf/sounds /home/rolf/tarballs /home/rolf/taxes /home/rolf/.bashrc /home/rolf/.enveloperc /home/rolf/.pan /home/rolf/.pan2 /home/rolf/Desktop /home/rolf/glabels /home/rolf/.kde4 /home/rolf/.glabels

Obviously, one path per line.

Postby **viking60** » 24 Jan 2011, 00:06

Great stuff with examples exactly what I need.
And that box really looks elegant. Firewire also has more speed than USB so no protests there. Dropbox is "cloud" and I am like Woody Allen: I don't trust clouds I can't see. So I'll drop that for now.
And you have also exclusively backed up from your home folder (even though the text file solution will allow any combination)?
Help me think here: My HD is gone and I have just installed Linux on the new HD. What do I need to obtain the old status quo (example :-D

but reality sooner or later )?
Anyway that code of yours is brilliant and it is a great idea to use a file that can be altered anytime for the backup. I took your code and mine and "merged" it:

Code: Select all

#!/bin/bash
for i in `cat viking.backup`; 
do rsync --delete -azvv -e  $i /media/My?Book/thomas-pc_backup/; done

And it works like a Million $ :dance:

For the sake of simplicity I removed the files to be excluded (for now). It does exactly the same as my code above.
I created the file viking.backup and put one line in it:

Code: Select all

/home/thomas

I think I can play around with this and even come up with something good. :tux5:

Keep it coming..... I am getting inspired here

Postby **rolf** » 24 Jan 2011, 00:46

To explain a little more, I've got a directory, /rsync, under my /home/rolf user directory. Here, I have created the rolf.backup text file list of paths to feed the bash script, and the bash script, which I have named rolf.backup.rsync.command
I cd to that directory, mount the external drive on /backup, and do:

Code: Select all

[rolf@localhost rsync]$ sh ./rolf.backup.rsync.command

As I understand, sh tells bash we are going to be giving you a script. That script is just a text file with my bash command in it:

Code: Select all

[rolf@localhost rsync]$ ll rolf.backup.rsync.command
-rw-r--r-- 1 rolf rolf 183 2011-01-09 16:05 rolf.backup.rsync.command
[rolf@localhost rsync]$ cat rolf.backup.rsync.command
for i in `cat rolf.backup`; do rsync -avz --delete --exclude 'default/kykd3xj8.slt/Cache' --exclude 'seamonkey/w1o08opq.default/Cache' --exclude 'lan/handycam' $i /backup/rolf/; done

You can see this text file is not even executable; it just works, at least when called this way in bash.

I'll point out some of the syntax of this for-do script entails backslash ` around instructions that might be interpreted by bash in steps, rather than as I perceive them. the --exclude arguments are enclosed in the single tick ' instead. I would just develop what I needed by trial and error, substituting some less dangerous command, like cat, for rsync or rm or whatever might not have good results, until I was sure the outcome would be as desired.

As for restoring, I have not ever done so but would just rsync back what I needed to. For example, I would mount the backup partition and do something like

rsync -av /backup/rolf/ /home/rolf

I do believe, as my backups are only directories and files in my /home directory. See `man rsync` and note that ../rolf/ with trailing slash means to transfer contents of rolf, while ../rolf without trailing slash will transfer the root directory, rolf/ as well.

I like Dropbox. I noted a Mark asking about it on the Mandriva forum. I use kde, too, and can just copy files from konqueror into the Dropbox/ directory and its subdirectories to be able to link a graphic or mp3, like I do here. I have to open the nautilus-dropbox from task bar to get the public url, however, but that is not too hard to do. I use it for easier file sharing than I see elsewhere, nothing security critical.

Postby **viking60** » 24 Jan 2011, 00:49

Sorry I edited... But it does not matter; your explanation is very good.
OK Guru likes dropbox I just installed the dropbox nautilus thing how do I make it work (In Gnome of course)?
Regarding calling a text file from a script: That works fine in PHP too I call my Mysql connection data and config data from a text file. It does not have to be php.
But I did not think of it here - you did and that is brilliant.

I do believe, as my backups are only directories and files in my /home directory. See `man rsync` and note that ../rolf/ with trailing slash means to transfer contents of rolf, while ../rolf without trailing slash will transfer the root directory, rolf/ as well.

So in my case above /home/thomas I transfered the entire thomas directory

Postby **rolf** » 24 Jan 2011, 01:17

Yes, I forgot, the list can include any sensible path on your mounted filesystem, afaik, where you have read access, at least.

There are some things in /home I would not like to lose and /home en toto

Code: Select all

[rolf@localhost rsync]$ du -s /home/rolf
276G    /home/rolf

includes much that does not merit duplicating. :party

On the dropbox, I think I just started it the first time, it was in the menu or

there is an executable in /usr/bin.

After that, I think you will be asked to agree to the EULA and it will download the whole shebang from the evil cloud. :twisted:

There should be a wizard walk-through, I see a man page but have not read it; there is a help item in the right-click menu from the taskbar icon.
I think you will see the light. :starwars1

Postby **viking60** » 24 Jan 2011, 03:39

OK I will wait a bit with dropbox because I don't think it is healthy to use it as a backup medium (non critical files are quite the opposite of a backup). So I concentrate on minimizing the backup this should really help:

Code: Select all

#!/bin/bash
for i in `cat viking.backup`; 
do rsync --delete -azvv -e --exclude=/tmp  $i /media/My?Book/thomas-pc_backup/; 
done

In neutral langauage:

Edit: Hmm I am not to sure that --exclude works like it should the process runs "forever". Ill let it run for some more time and we will see...
Jupp this one mostly went through but with an error message:

That error also occurs when using the single tick ' method :confused

Ah I have a theory why the exclude is not working - It must be the file system! That is why the already created folders cannot be deleted (maybe).
I also get an I/O error so this is my problem:

Postby **viking60** » 24 Jan 2011, 16:24

OK this is as far as I have come:

It works but I get this error:

I have made a text file called ex.txt and the content of it is:

It contains all I want to exclude.

Postby **rolf** » 24 Jan 2011, 18:13

I'm on the Treo and it won't show spoiler content. So, I can only add that --exclude doesn't delete what is already on the backup, here, either and to check the path of your --exclude arguments. I think there is a statement about this in the man or maybe I saw it @ google?

Postby **viking60** » 24 Jan 2011, 19:05

I think this looked promising. But it does not matter how I exclude. In fact I believe there are a lot out there that actually think that things are excluded, while in fact they are not.
Ahrg this is frustrating.
And you are on Treo - is that legal?

Postby **rolf** » 24 Jan 2011, 19:20

Treo 755p, Palm Blazer v. 4.5, Sprint SERO plan w/ unlimited data, text, 500 minutes: $30/mo., ftw

Postby **viking60** » 24 Jan 2011, 19:47

Ah that must be the formula from the drug store :lol:

Nice
And finally I got the --exclude thing working!! I installed lib64rsync1 and lib64rsync-devel :Doh:

As they say: Today we stand at the edge of the cliff, but tomorrow we could be one step further....
Working on an testing like h.....

bjoernvold.com

Lets make a killer backup tool

Lets make a killer backup tool

Re: Lets make a killer backup tool

Re: Lets make a killer backup tool

Re: Lets make a killer backup tool

Re: Lets make a killer backup tool

Re: Lets make a killer backup tool

Re: Lets make a killer backup tool

Re: Lets make a killer backup tool

Re: Lets make a killer backup tool

Re: Lets make a killer backup tool

Re: Lets make a killer backup tool

Re: Lets make a killer backup tool