The Linux Leap

[viking60]

Short answer to all your questions:
Yes!

You would only add /etc/apache2/ to viking.backup in the example above and so on.

[Snorkasaurus]

One of the problems I had was that after /etc/apache2/ is complete, I then want to do /etc/dnsmasq.d/ - but rsync did not like the fact that /etc/apache2/ is already at the destination, so it deleted it. And so on. Even if I was doing it incorrectly, the lack of support for non-standard SSH servers is a killer for me.
S.

[viking60]

Rsync does synchronize. So you get what you backup - exactly. If there is something at the destination then rsync will alter it so that it matches what you back up.

[Snorkasaurus]

Rsync does synchronize. So you get what you backup - exactly. If there is something at the destination then rsync will alter it so that it matches what you back up.

What would the script look like to make the destination backup directory look like this:

I also believe I recall that rsync had problems with backing up servers through a tunnel that is on a non-standard port (ie: create a tunnel to Machine A on port 12345 and then connect to Machine B through that tunnel).

S.

[Snorkasaurus]

So I figured I would try the script and see what it does. I think that part of the problem is that I am not doing backups to a local device (see image) and I need to limit the transfer speed. Perhaps a better explanation is necessary.

• Both routers are Debian Wheezy boxes using iptables scripts and masq'ing
• The router on "network 1" has no open ports, NAT, or PAT to the outside world.
• The router on "network 2" has port 12345 forwarded to 22 on Linux box "pc 3"
• sshd on "pc 3" allows only key based authentication
• "pc 1" is the Linux box that would be doing the backups
• "pc 2" and "pc 4" are Windows XP boxes running FreeSSHd
• "pc 3" is a Wheezy box
• "pc 5" is a Windows Server 2003 box running FreeSSHd

I wish to backup the configuration and data of "pc 2" through "pc 5" and both routers, to "pc1" and push the configuration of "pc 1" on to "pc 3". I wish to only download files that are newer, and I wish to limit the transfer speed to 32kBps when downloading or uploading across the Internet (not required for intra-network transfers).

The reason I mentioned /etc directories previously is because in many cases it seems backup solutions have a problem with selecting individual items within a directory while ignoring others. So the backup solution would backup /etc/apache2/ in one step and then the next step says to backup /etc/dnsmasq.d/ - but the destination directory already has "apache2" in the /etc directory, so it deletes it (because the instructions in step 2 said to NOT transfer anything except /etc/dnsmasq.d. I believe your script may be able to deal with this because it uses a viking.backup file with a specific list of selections that are likely all read before the transfer starts.

rsync seems to have a bwlimit argument that I think I had trouble with. Possibly because a number of the files are small and do not trigger it.

And then there is the FreeSSHd problem.

S.

[viking60]

Rsync leaves the files on the target that are identical to those you backup and adds new files and replaces files that are changed - it is synchronization.
The backup target can be reached via ssh on Linux - I have not used it with Windows since I rarely use Windows other than in Virtualbox under Linux.

[Snorkasaurus]

Rsync leaves the files on the target that are identical to those you backup and adds new files and replaces files that are changed - it is synchronization.

I know what it does, what I thought might be a problem is having more than one item being backed up to the same place with "purging" turned on. As stated, some backup solutions will delete previously backed up data each time because it is "extra" as far as the new backup task is concerned. Using the for loop in the batch file does not do this as rsync is smart enough to gather all the information first before starting. Here's a chronological way of looking at it:
Step 1: Backup remote:/etc/apache2/ to /backups/server/etc - and delete files which do not exist at the source.

This does what it says.

Step 2: Backup remote:/etc/dnsmasq.d./ to /backups/server/etc - and delete files which do not exist at the source.

In this case, SOME backup solutions will backup /etc/dnsmasq.d/ and then DELETE /backups/server/etc/apache2 because it is not part of the source (which is only /etc/dnsmasq.d/).

The for loop in your script is apparently smart enough (well, rsync is smart enough) to collect all required information before copying so it does not do this, but there are some backup solutions that do.

The backup target can be reached via ssh on Linux

What I mean is backing up a machine that you have no direct access to... SSH'ing to one machine on another network, and then SSH'ing from that machine to a subsequent machine to do the rsync copy. In the image above, network 1 has no SSH access to "router", "pc 4", or "pc 5" and must access them by first creating an SSH tunnel to "pc 3". This limitation means that any machine on network 2 that is going to be backed up, needs to have direct SSH access from the outside world.

I have not used it with Windows since I rarely use Windows other than in Virtualbox under Linux.

I'll have to retry this tomorrow.

By the way, I believe the "r" in your "rsync -azrv" within your script can be left out since "r" is implied by "a".

S.

[viking60]

Rsync leaves the files on the target that are identical to those you backup and adds new files and replaces files that are changed - it is synchronization.

I know what it does, what I thought might be a problem is having more than one item being backed up to the same place with "purging" turned on. As stated, some backup solutions will delete previously backed up data each time because it is "extra" as far as the new backup task is concerned. Using the for loop in the batch file does not do this as rsync is smart enough to gather all the information first before starting. Here's a chronological way of looking at it:
Step 1: Backup remote:/etc/apache2/ to /backups/server/etc - and delete files which do not exist at the source.

This does what it says.

Yes it does

Step 2: Backup remote:/etc/dnsmasq.d./ to /backups/server/etc - and delete files which do not exist at the source.

In this case, SOME backup solutions will backup /etc/dnsmasq.d/ and then DELETE /backups/server/etc/apache2 because it is not part of the source (which is only /etc/dnsmasq.d/).

The for loop in your script is apparently smart enough (well, rsync is smart enough) to collect all required information before copying so it does not do this, but there are some backup solutions that do.

You can leave old backuped stuff on the backup media and only add new or changed stuff. This would not be true synchronization though since that pretty much demands that the content on the source and the backup medium is the same. Rsync has a lot of options so you can tailor it to do that though.
One of the problems I had to solve was to get rsync to delete earlier backed up stuff that is no longer on the source. In this case that is what you want; so it would not be a problem for you.

By the way, I believe the "r" in your "rsync -azrv" within your script can be left out since "r" is implied by "a".

S.

Well that is the one that worked the -r ensures that subdirectories are synced too. the -a stands for archive and equals -rlptgoD; it might be covered but I am not sure if it is to a 100% (I think you are right about that though).

In any case you could start with this easy setup and test script. It will prompt you and create all the necessary files (not good for a cronjob though since you do not want to be prompted there).
gbackup.sh

Code: Select all

#!/bin/bash
#Setting up some colors I might use later here:
red='\e[0;31m'
RED='\e[1;31m'
blue='\e[0;34m'
BLUE='\e[1;34m'
cyan='\e[0;36m'
CYAN='\e[1;36m'
NC='\e[0m' # No Color
echo -e "Hi, please type $CYAN/home/<your name>$NC here (your home directory or whatever): \c "
read  hjem
echo -e "Hi, please type the $CYAN/full/path/to_the target/$NC here (where your backup will be stored): \c "
read maal
cd $hjem
# checking if ex.txt exists
if [ -e 'ex.txt' ]
then
echo "ex.txt exists"
else
#Creating ex.txt and populating it with /home/<my_name>/.gvfs to avoid errors when running as root
echo '#Here you can put the all the files and directories you do not want to backup. Like the example below:'>>ex.txt
echo '*/.gvfs'>>ex.txt
echo "ex.txt has been created"
fi
#Creating viking.backup and populating it with the home directory and /etc (must run as root!)
if [ -e 'viking.backup' ]
then
echo "viking.backup exists"
else
echo $hjem>>viking.backup #Creates viking.backup and putting </home/your_name> in there
echo '/etc'>>viking.backup #adding the /etc directory to be backed up
fi
#The Magic happens!
for i in $(cat viking.backup);
do rsync -azrv --delete --delete-excluded  --exclude-from 'ex.txt'  $i $maal;
#Creating a log (backup.log) to se when the last backup was run - nice when checking cron jobs
echo -n 'Last backup at the bottom:'>>backup.log|date +'%d-%m-%Y klokken %H:%M'>>backup.log
done


Once all the files are created you can edit ex.txt and viking.backup to fine tune what to back up and what to exclude.
If you do not want to back up your home directory you simply enter another path on the first prompt (or edit viking.backup later).
The date and time of every backup will be saved in backup.log.

[viking60]

Regarding SSH; here are a few things that you can check out (I have checked these).

Don't run FreeSSHd as a daemon in Windows- it is buggy (doesn't work here).
Install it and start it from the GUI - that works. This will not work if you have set FreeSSH to run as a daemon though. In that case remove it and re-install without picking the set as daemon option.

Then I logged in via a terminal from Linux the first time to get possible key error and man in the middle attack messages. If that is the case then you already have a key for that Computer in your ~/.ssh/known_hosts that does not match.

Here you simply edit ~/.ssh/known_hosts and remove the line that starts with the IP or the name.
then ssh back in and accept the connection with a simple "yes".

After that you are good to go from a file manager here I am logged in to a windows 7 laptop with ssh (it is automatically converted to sftp from Thunar here)

Edit:
Hmm I just tried to pull a directory from the Windows box to my Linux box and ssh failed with an incompatibility problem

Code: Select all

protocol version mismatch -- is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at compat.c(176) [Receiver=3.1.1]


I don't know if there is an rsync version for Windows since it probably needs to be on both ends....
Checking out Delta Copy ..and that didn't work
Looks like I will have to do it the Cygwin way...

[Snorkasaurus]

You can leave old backuped stuff on the backup media and only add new or changed stuff.

Hmmm, I don't think I am explaining it very well. What I mean is that some backup solutions consider each selection (ie: each line of viking.backup) to be a separate task. So it might backup /etc/apache2/ first, but it deletes that when it backs up /etc/dnsmasq.d/... which gets deleted when it backs up /etc/postfix/... which gets deleted when it backs up /etc/whatever. The reason it does this, is because I use the same destination such as "/backup/server/etc/" and the purging mechanism is not smart enough to build a complete list of all items to be backed up first, rather than backing each item up separately.

In any case you could start with this easy setup and test script.

I am back at it this morning and it is generally working, though I had to add a "-e" to make the connection first of course.

Code: Select all

rsync -azrv -e "ssh -p 1234 -i /path/rsa-key" --delete --delete-excluded  --exclude-from 'ex.txt' root@server:$i /backups/server/;

I did notice however that if I leave the comment at the top of the viking.backup the script barfs all over it.

Now I have to see if the bwlimit switch works or if I will have to screw with trickle to throttle bandwidth.

S.