Lets make a killer backup tool

What do you have and what do you want?

Moderators: b1o, jkerr82508

User avatar
viking60
Über-Berserk
Posts: 9344
Joined: 14 Mar 2010, 16:34

Lets make a killer backup tool

Postby viking60 » 23 Jan 2011, 20:10

I am a fairly simple Magazine reader so I never understood all those fancy backup solutions out there. Mandriva had some crappy daemons and drakbackup does fill up your HD to the point where Linux gets unresponsive.

So I had a look at Bacula - Yeah right :shock: I went for a Whiskey after 10 minutes and dropped it.
(It seems to be a great tool so it is me being unfair and impatient here).
And even If I understood how to operate different backup programs I never fully understood how they operated. To the point that I did not know where to look for files for restoring.
And I see all those competent experts out there talking about backups and I get the impression they do it every day with ease and total control. That leaves sloppy me!
Well actually I have found a way that I can control: I copy my home directory to an external USB Harddisk (My Book) and I know it works because I have restored it when necessary.
So that works - kind of. And should disaster strike I can always reinstall the distro and All the programs and create my User (puh..) restore the home directory and all is back.

It can be done a lot easier than that you say? Yes it can, and I will not be knowing what I am doing - and I hate that. :-x
But back to the simple aproach and this is where you gurus can be really useful:

I want this thread to end with a script or program that we have made here (OK I can do all the scripting but I prefer not to).
So what do we need to back up to make a sensible restore when disaster strikes? (Not all of it! Reinstalling the distro is fine).
What media to use? Can we use dropbox
How can we make use of cron jobs?
Network solutions -ssh?
Anything else?


SInce Berserks are brutal and undemocratic I'll start and dictate that we must make use of rsync. I have not made a script I just write this line every time:

Code: Select all

rsync --delete -azvv -e  /home/thomas /media/My?Book/thomas-pc_backup

This backs up my entire home directory (thomas) to the attached My Book in a folder I have named thomas-pc_backup.
It works fine I can put back whatever I want from it. This is useful when I have destroyed Virtual box installations etc.

When I have My Book attached to to my server I make use of ssh:

Code: Select all

rsync --delete -azvv -e ssh /home/thomas thomas@viking60-server:/media/My?Book/thomas-pc_backup

And start it from the same box. The only differense is that the My Book is attached to the server.
That works fine to. Can we think of a solution with dropbox here?

As you can see I need to automate this stuff in a script and I thought I'll add some fancy stuff that you have helped me with.
What other areas should be backed up? Lets start there.
The stage is yours :A

PS if you want to go directly to the succesfull conclusion (pt) you can go here
Or you can download it from github
You will find the wiki here
Manjaro 64bit on the main box -Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz and nVidia Corporation GT200b [GeForce GTX 275] (rev a1. + Centos on the server - Arch on the laptop.
"There are no stupid questions - Only stupid answers!"

User avatar
rolf
Guru-Berserk
Posts: 1107
Joined: 16 Mar 2010, 16:07

Re: Lets make a killer backup tool

Postby rolf » 23 Jan 2011, 22:54

I'll just throw a few strands of what I do against the wall, see if any of it sticks. Maybe you can make use of some bits.

  • 1TB Seagate SATA drive in an external Icy Dock enclosure, USB2 or Firewire connectors; I'm using Firewire. Image
  • Mount point, /backup, that I make owned by my normal user, so not to have to be root to do backups.
  • Icy Dock is always connected, can be turned on/off with a power switch. When I'm ready to do a backup, I watch the following output in konsole as I turn on the switch:

    Code: Select all

    $ tailf /var/log/messages
    Jan 23 13:41:30 localhost kernel: scsi7 : SBP-2 IEEE-1394
    Jan 23 13:41:31 localhost kernel: ieee1394: sbp2: Logged into SBP-2 device
    Jan 23 13:41:31 localhost kernel: ieee1394: sbp2: Node 0-00:1023: Max speed [S400] - Max payload [2048]
    Jan 23 13:41:31 localhost kernel: scsi 7:0:0:0: Direct-Access     ST310005 28AS             CC35 PQ: 0 ANSI: 4
    Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: Attached scsi generic sg8 type 0
    Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
    Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Write Protect is off
    Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Cache data unavailable
    Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Assuming drive cache: write through
    Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Cache data unavailable
    Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Assuming drive cache: write through
    Jan 23 13:41:31 localhost kernel: sdg: sdg1 sdg2 < sdg5 >
    Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Cache data unavailable
    Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Assuming drive cache: write through
    Jan 23 13:41:31 localhost kernel: sd 7:0:0:0: [sdg] Attached SCSI disk

    and, knowing the appropriate partition, I will become root in order to mount it, something like:

    Code: Select all

    # mount /dev/sdg1 /backup
  • Finally, a little bash script I massaged into life, with Google's help, many moons ago.

    Code: Select all

    for i in `cat rolf.backup`; do rsync -avz --delete --exclude 'default/kykd3xj8.slt/Cache' --exclude 'seamonkey/w1o08opq.default/Cache' --exclude 'lan/handycam' $i /backup/rolf/; done


    where /rolf is a directory I created at the root of the partition on the Seagate.

    My copy of `man rsync` has always had a different syntax for --exclude and it hasn't ever worked for me. Faced with 24G of lan's handycam now on disk in the path of my backup scheme, I googled what is there, now, and it seems correct. This is crude, a work in progress, but good enough for my current purposes. Here, in the same directory as I issue the bash command, I keep a text file with the paths I want to archive, rolf.backup

    Code: Select all

    [rolf@localhost rsync]$ cat rolf.backup
    /home/rolf/.gftp
    /home/rolf/taxes
    /home/rolf/.mozilla
    /home/rolf/addresses
    /home/rolf/documents
    /home/rolf/envelope
    /home/rolf/humor
    /home/rolf/info
    /home/rolf/jobs
    /home/rolf/maps
    /home/rolf/movie
    /home/rolf/mp3
    /home/rolf/paul
    /home/rolf/personal
    /home/rolf/pictures
    /home/rolf/pmfax
    /home/rolf/rpm
    /home/rolf/sounds
    /home/rolf/tarballs
    /home/rolf/taxes
    /home/rolf/.bashrc
    /home/rolf/.enveloperc
    /home/rolf/.pan
    /home/rolf/.pan2
    /home/rolf/Desktop
    /home/rolf/glabels
    /home/rolf/.kde4
    /home/rolf/.glabels

    Obviously, one path per line.
:A

User avatar
viking60
Über-Berserk
Posts: 9344
Joined: 14 Mar 2010, 16:34

Re: Lets make a killer backup tool

Postby viking60 » 24 Jan 2011, 00:06

Great stuff with examples exactly what I need.
And that box really looks elegant. Firewire also has more speed than USB so no protests there. Dropbox is "cloud" and I am like Woody Allen: I don't trust clouds I can't see. So I'll drop that for now.
And you have also exclusively backed up from your home folder (even though the text file solution will allow any combination)?
Help me think here: My HD is gone and I have just installed Linux on the new HD. What do I need to obtain the old status quo (example :-D but reality sooner or later )?
Anyway that code of yours is brilliant and it is a great idea to use a file that can be altered anytime for the backup. I took your code and mine and "merged" it:

Code: Select all

#!/bin/bash
for i in `cat viking.backup`;
do rsync --delete -azvv -e  $i /media/My?Book/thomas-pc_backup/; done

And it works like a Million $ :dance: For the sake of simplicity I removed the files to be excluded (for now). It does exactly the same as my code above.
I created the file viking.backup and put one line in it:

Code: Select all

/home/thomas

I think I can play around with this and even come up with something good. :tux5:
Keep it coming..... I am getting inspired here
Manjaro 64bit on the main box -Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz and nVidia Corporation GT200b [GeForce GTX 275] (rev a1. + Centos on the server - Arch on the laptop.
"There are no stupid questions - Only stupid answers!"

User avatar
rolf
Guru-Berserk
Posts: 1107
Joined: 16 Mar 2010, 16:07

Re: Lets make a killer backup tool

Postby rolf » 24 Jan 2011, 00:46

To explain a little more, I've got a directory, /rsync, under my /home/rolf user directory. Here, I have created the rolf.backup text file list of paths to feed the bash script, and the bash script, which I have named rolf.backup.rsync.command
I cd to that directory, mount the external drive on /backup, and do:

Code: Select all

[rolf@localhost rsync]$ sh ./rolf.backup.rsync.command


As I understand, sh tells bash we are going to be giving you a script. That script is just a text file with my bash command in it:

Code: Select all

[rolf@localhost rsync]$ ll rolf.backup.rsync.command
-rw-r--r-- 1 rolf rolf 183 2011-01-09 16:05 rolf.backup.rsync.command
[rolf@localhost rsync]$ cat rolf.backup.rsync.command
for i in `cat rolf.backup`; do rsync -avz --delete --exclude 'default/kykd3xj8.slt/Cache' --exclude 'seamonkey/w1o08opq.default/Cache' --exclude 'lan/handycam' $i /backup/rolf/; done


You can see this text file is not even executable; it just works, at least when called this way in bash.

I'll point out some of the syntax of this for-do script entails backslash ` around instructions that might be interpreted by bash in steps, rather than as I perceive them. the --exclude arguments are enclosed in the single tick ' instead. I would just develop what I needed by trial and error, substituting some less dangerous command, like cat, for rsync or rm or whatever might not have good results, until I was sure the outcome would be as desired.

As for restoring, I have not ever done so but would just rsync back what I needed to. For example, I would mount the backup partition and do something like

rsync -av /backup/rolf/ /home/rolf

I do believe, as my backups are only directories and files in my /home directory. See `man rsync` and note that ../rolf/ with trailing slash means to transfer contents of rolf, while ../rolf without trailing slash will transfer the root directory, rolf/ as well.

I like Dropbox. I noted a Mark asking about it on the Mandriva forum. I use kde, too, and can just copy files from konqueror into the Dropbox/ directory and its subdirectories to be able to link a graphic or mp3, like I do here. I have to open the nautilus-dropbox from task bar to get the public url, however, but that is not too hard to do. I use it for easier file sharing than I see elsewhere, nothing security critical. ;)

User avatar
viking60
Über-Berserk
Posts: 9344
Joined: 14 Mar 2010, 16:34

Re: Lets make a killer backup tool

Postby viking60 » 24 Jan 2011, 00:49

Sorry I edited... But it does not matter; your explanation is very good.
OK Guru likes dropbox I just installed the dropbox nautilus thing how do I make it work (In Gnome of course)?
Regarding calling a text file from a script: That works fine in PHP too I call my Mysql connection data and config data from a text file. It does not have to be php.
But I did not think of it here - you did and that is brilliant.
I do believe, as my backups are only directories and files in my /home directory. See `man rsync` and note that ../rolf/ with trailing slash means to transfer contents of rolf, while ../rolf without trailing slash will transfer the root directory, rolf/ as well.
+1
So in my case above /home/thomas I transfered the entire thomas directory
Manjaro 64bit on the main box -Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz and nVidia Corporation GT200b [GeForce GTX 275] (rev a1. + Centos on the server - Arch on the laptop.
"There are no stupid questions - Only stupid answers!"

User avatar
rolf
Guru-Berserk
Posts: 1107
Joined: 16 Mar 2010, 16:07

Re: Lets make a killer backup tool

Postby rolf » 24 Jan 2011, 01:17

Yes, I forgot, the list can include any sensible path on your mounted filesystem, afaik, where you have read access, at least.

There are some things in /home I would not like to lose and /home en toto

Code: Select all

[rolf@localhost rsync]$ du -s /home/rolf
276G    /home/rolf


includes much that does not merit duplicating. :party

On the dropbox, I think I just started it the first time, it was in the menu or

Code: Select all

$ rpm -ql nautilus-dropbox
/usr/bin/dropbox
/usr/lib64/nautilus/extensions-2.0/libnautilus-dropbox.so
/usr/share/applications/dropbox.desktop
/usr/share/icons/hicolor/16x16/apps/dropbox.png
/usr/share/icons/hicolor/22x22/apps/dropbox.png
/usr/share/icons/hicolor/24x24/apps/dropbox.png
/usr/share/icons/hicolor/32x32/apps/dropbox.png
/usr/share/icons/hicolor/48x48/apps/dropbox.png
/usr/share/icons/hicolor/64x64/apps/dropbox.png
/usr/share/icons/hicolor/64x64/emblems/emblem-dropbox-syncing.png
/usr/share/icons/hicolor/64x64/emblems/emblem-dropbox-unsyncable.png
/usr/share/icons/hicolor/64x64/emblems/emblem-dropbox-uptodate.png
/usr/share/man/man1
/usr/share/man/man1/dropbox.1.lzma
there is an executable in /usr/bin.

After that, I think you will be asked to agree to the EULA and it will download the whole shebang from the evil cloud. :twisted:

There should be a wizard walk-through, I see a man page but have not read it; there is a help item in the right-click menu from the taskbar icon.
I think you will see the light. :starwars1

User avatar
viking60
Über-Berserk
Posts: 9344
Joined: 14 Mar 2010, 16:34

Re: Lets make a killer backup tool

Postby viking60 » 24 Jan 2011, 03:39

OK I will wait a bit with dropbox because I don't think it is healthy to use it as a backup medium (non critical files are quite the opposite of a backup). So I concentrate on minimizing the backup this should really help:

Code: Select all

#!/bin/bash
for i in `cat viking.backup`;
do rsync --delete -azvv -e --exclude=/tmp  $i /media/My?Book/thomas-pc_backup/;
done

In neutral langauage:

Code: Select all

#!/bin/bash
for i in `cat viking.backup`;
do rsync --delete -azvv -e --exclude=/tmp  $i /media/<YOUR EXTERNAL HD>/<YOUR BACKUP DIRECTORY>/;
done

Edit: Hmm I am not to sure that --exclude works like it should the process runs "forever". Ill let it run for some more time and we will see...
Jupp this one mostly went through but with an error message:

Code: Select all

sent 27839716918 bytes  received 759988 bytes  6348301.65 bytes/sec
total size is 260869537475  speedup is 9.37
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7]

That error also occurs when using the single tick ' method :confused
Ah I have a theory why the exclude is not working - It must be the file system! That is why the already created folders cannot be deleted (maybe).
I also get an I/O error so this is my problem:
From the man page description of --delete

If the sending side detects any I/O errors, then the deletion of
any files at the destination will be automatically disabled.
This is to prevent temporary filesystem failures (such as NFS
errors) on the sending side from causing a massive deletion of
files on the destination. You can override this with the
--ignore-errors option.
Manjaro 64bit on the main box -Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz and nVidia Corporation GT200b [GeForce GTX 275] (rev a1. + Centos on the server - Arch on the laptop.
"There are no stupid questions - Only stupid answers!"

User avatar
viking60
Über-Berserk
Posts: 9344
Joined: 14 Mar 2010, 16:34

Re: Lets make a killer backup tool

Postby viking60 » 24 Jan 2011, 16:24

OK this is as far as I have come:

Code: Select all

#!/bin/bash
for i in `cat viking.backup`;
do rsync -azt --delete-excluded -e --exclude-from 'ex.txt'  $i /media/My?Book/thomas-pc_backup/;
done

It works but I get this error:

Code: Select all

rsync: opendir "/home/thomas/.config/gsmartcontrol" failed: Permission denied (13)
IO error encountered -- skipping file deletion
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7]

I have made a text file called ex.txt and the content of it is:

Code: Select all

tmp/
/.config/gsmartcontrol

It contains all I want to exclude.
Manjaro 64bit on the main box -Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz and nVidia Corporation GT200b [GeForce GTX 275] (rev a1. + Centos on the server - Arch on the laptop.
"There are no stupid questions - Only stupid answers!"

User avatar
rolf
Guru-Berserk
Posts: 1107
Joined: 16 Mar 2010, 16:07

Re: Lets make a killer backup tool

Postby rolf » 24 Jan 2011, 18:13

I'm on the Treo and it won't show spoiler content. So, I can only add that --exclude doesn't delete what is already on the backup, here, either and to check the path of your --exclude arguments. I think there is a statement about this in the man or maybe I saw it @ google?

User avatar
viking60
Über-Berserk
Posts: 9344
Joined: 14 Mar 2010, 16:34

Re: Lets make a killer backup tool

Postby viking60 » 24 Jan 2011, 19:05

I think this looked promising. But it does not matter how I exclude. In fact I believe there are a lot out there that actually think that things are excluded, while in fact they are not.
Ahrg this is frustrating.
And you are on Treo - is that legal? :P
Manjaro 64bit on the main box -Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz and nVidia Corporation GT200b [GeForce GTX 275] (rev a1. + Centos on the server - Arch on the laptop.
"There are no stupid questions - Only stupid answers!"

User avatar
rolf
Guru-Berserk
Posts: 1107
Joined: 16 Mar 2010, 16:07

Re: Lets make a killer backup tool

Postby rolf » 24 Jan 2011, 19:20

:confused
Treo 755p, Palm Blazer v. 4.5, Sprint SERO plan w/ unlimited data, text, 500 minutes: $30/mo., ftw

User avatar
viking60
Über-Berserk
Posts: 9344
Joined: 14 Mar 2010, 16:34

Re: Lets make a killer backup tool

Postby viking60 » 24 Jan 2011, 19:47

Ah that must be the formula from the drug store :lol:
Image

Nice
And finally I got the --exclude thing working!! I installed lib64rsync1 and lib64rsync-devel :Doh:
As they say: Today we stand at the edge of the cliff, but tomorrow we could be one step further....
Working on an testing like h.....
Manjaro 64bit on the main box -Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz and nVidia Corporation GT200b [GeForce GTX 275] (rev a1. + Centos on the server - Arch on the laptop.
"There are no stupid questions - Only stupid answers!"


Return to “Software”