All About Linux Backups with rsync

all about tech.
Post Reply
george
Site Admin
Posts: 13
Joined: Thu Aug 01, 2019 3:16 am
All About Linux Backups with rsync

Post by george » Wed Sep 09, 2020 4:24 pm

If you are attempting to copy your entire /var folder to an external drive using rsync, but you run into issues with rsync like this:

Code: Select all

sent 169,748,362,821 bytes  received 3,982,112 bytes  16,898,347.01 bytes/sec
total size is 193,085,110,161  speedup is 1.14
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0] 
the best course of action is to NOT copy the entire /var folder, but instead only copy the /var/www folder. Using rsync with /var/lib is usually where the files fail to transfer, to circumvent this you can use the mysql dump command which will backup all databases to one file. Here is an example of my backup process for my linux webserver

Code: Select all

Backup protocol:
sudo fdisk -l
mount /dev/sdb1 /mnt
mysqldump -u root -p --all-databases > alldbs.sql
mv alldbs.sql /mnt/backup
rsync -av /var /mnt/backup (will likely fail to copy entire folder)
rsync -av /var/www /mnt/backup 
rsync -av /home /mnt/backup
rsync -av /etc /mnt/backup
umount -l /dev/sdb1 

Above command descriptions, in order:
lists all disks on system
mounts external drive (sdb1 - [partition of sdb] could be different)
dumps all mysql databases to a file called alldbs.sql
moves all databases file onto backup drive
copies almost the entire var directory into the drive backup folder
copies entire var/www web directory into the drive backup folder
copies entire home directory into the drive backup folder
copies entire etc directory into the drive backup folder
unmounts the external drive (sdb1)  [using -l because the filesystem is still accessing the /mnt folder, but everything is already copied]
rsync flags options explained:

Code: Select all

-a: archive mode: preserves file permissions, recursively copies, copies symlinks, preserves modification times, preserves groups, preserves file ownership, preserves device files
-v: verbose mode: displays which files are being copied 
If you just wish to have the data transfered as ASCII rather than binary, you can use the rsync flag option -h, which stands for "human readable." This is not recommended at all...

Now lets talk filesystem format of the exteral drive. Preferably, you want to use ext4 to preserve all permissions on the linux system, but you can alternatively use NTFS if you plan on modifying the permissions later (but certain executables could have problems):

Safety aside (which as other users have established should unlikely be an issue), you must keep in mind that the permission system on NTFS and ext4 is different and cannot be "carried over" from one file system to another without loss of information.

For instance, if you mark a script as "executable" on your main ext4 drive and then copy it to the NTFS backup and back, it will no longer be marked as such. Similarly, files marked as "read only" will become accessible to everyone if you mount the NTFS partition with the standard 644 file permissions (i.e. everyone can read the file, but only the owner can write into it).

All this means that while restoring media like videos or photos from a backup should be absolutely fine, restoring an app may require you to tweak permissions and ownership for its executables. Likewise, if you restore a file that was meant to be read only by you (400), everyone will be able to read it unless you remember to re-set its permissions.

One workaround to the problem above might be be putting all your backed-up data into a UNIX-native container such as tar that will preserve such information. Doing incremental backups with tar is difficult, however, and you may end up with the same file taking hard drive space in multiple tarred snapshots.

Considering the above, I would suggest formatting the drive to your system-native file system, in your case ext4, backing up your data with rsync, and reformatting it back to NTFS if you need to repurpose that drive later on for Windows.

If you feel a little bit adventurous, you may want to take a look at the Btr(fs) file system due to its built-in copy-on-write optimization and snapshots that may come in handy for backups. Last time I checked it had issues, but also looked very promising.


Post Reply