The other day I needed to create a server that would be responsible for the backing up of data on a Windows share.
This was a critical chore since this share contained every department's data of said company.
Deduplication was an obvious choice for the disk space savings. Also, a method of syncing that would allow a differential backup had to be employed as well.
The obvious choice for deduplication would be ZFS, which is the most popular one. Alas, it is a memory hog and therefore Lessfs and OpenDedup were considered instead. In the end, I chose OpenDedup, as in the time of writing this, it was a project that was more recently updated than its competitor.
The obvious choice for the actual backing up of the files was rsync. No other comments are required, I think.
CentOS 6.8 was chosen as the O/S of choice for this one.
Our IP is 192.168.1.36 and the Windows share is at \\192.168.0.29\WORK. Our domain is MYNETWORK and the domain administrator's password is p@$$W0r(!
So let's move on to the deployment. Updating our packages and installing our favorite repos for one:
Now, installing necessary packages:
Configure ntp:
Disable SELinux:
We'll be taking the backup on an external USB drive. Let's see what's already mounted:
And the disks available are:
There were a multitude of ways we could find this out, but I find the one in particular the easiest and fastest one.
Let's format our disk:
This was a critical chore since this share contained every department's data of said company.
Deduplication was an obvious choice for the disk space savings. Also, a method of syncing that would allow a differential backup had to be employed as well.
The obvious choice for deduplication would be ZFS, which is the most popular one. Alas, it is a memory hog and therefore Lessfs and OpenDedup were considered instead. In the end, I chose OpenDedup, as in the time of writing this, it was a project that was more recently updated than its competitor.
The obvious choice for the actual backing up of the files was rsync. No other comments are required, I think.
CentOS 6.8 was chosen as the O/S of choice for this one.
Our IP is 192.168.1.36 and the Windows share is at \\192.168.0.29\WORK. Our domain is MYNETWORK and the domain administrator's password is p@$$W0r(!
So let's move on to the deployment. Updating our packages and installing our favorite repos for one:
[root@BAK ~]# yum -y update
[root@BAK ~]# rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
[root@BAK ~]# rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
[root@BAK ~]# vi /etc/yum.repos.d/epel.repo[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 6 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch/debug
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-6&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
gpgcheck=1
[epel-source]
name=Extra Packages for Enterprise Linux 6 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/6/SRPMS
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-6&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
gpgcheck=1
Now, installing necessary packages:
[root@BAK ~]# yum -y install autofs cifs-utils smbclient samba-client samba samba-common samba-doc rsync ntp ntpdateConfigure ntp:
[root@BAK ~]# chkconfig ntpd on
[root@BAK ~]# ntpdate -d -u 3.europe.pool.ntp.org
[root@BAK ~]# service ntpd startDisable SELinux:
[root@BAK ~]# echo 0 > /selinux/enforce
[root@BAK ~]# vi /etc/sysconfig/selinux# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
We'll be taking the backup on an external USB drive. Let's see what's already mounted:
[root@BAK ~]# df -khFilesystem Size Used Avail Use% Mounted on
/dev/sdd3 909G 3.5G 859G 1% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sdd1 512M 280K 512M 1% /boot/efi
/dev/sde1 147G 60M 140G 1% /export
And the disks available are:
[root@BAK ~]# blkid -c /dev/null/dev/sde1: UUID="0a9cb7c1-ee2c-439b-a73e-c55441f9743a" TYPE="ext4"
/dev/sdd1: SEC_TYPE="msdos" UUID="B7C5-062C" TYPE="vfat"
/dev/sdd2: UUID="98591ed9-25a1-404c-be7f-7740a53f69c7" TYPE="swap"
/dev/sdd3: UUID="b1724545-924d-4ec1-814b-73301637713a" TYPE="ext4"
/dev/sdf1: UUID="5dd578a1-b77c-4d3f-b21a-8f330474667a" TYPE="ext4"
There were a multitude of ways we could find this out, but I find the one in particular the easiest and fastest one.
Let's format our disk:
[root@BAK ~]# parted /dev/sdfGNU Parted 2.1
Using /dev/sdf
Welcome to GNU Parted! Type 'help' to view a list of commands.(parted) mklabel gptWarning: The existing disk label on /dev/sdf will be destroyed and all data on this disk will be lost. Do you want to continue?Yes/No? yes
(parted) unit TB
(parted) mkpart primary 0.00TB 4.00TB
(parted) quitInformation: You may need to update /etc/fstab.[root@BAK ~]# mkfs.ext4 /dev/sdf1mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1 blocks, Stripe width=0 blocks
244187136 inodes, 976745728 blocks
48837286 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
29808 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Mount it and make it a permanent mount:
[root@BAK ~]# mkdir /work_backup
[root@BAK ~]# mount /dev/sdf1 /work_backup
[root@BAK ~]# df -khFilesystem Size Used Avail Use% Mounted on
/dev/sdd3 909G 3.5G 859G 1% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sdd1 512M 280K 512M 1% /boot/efi
/dev/sde1 147G 60M 140G 1% /export
/dev/sdc1 3.6T 68M 3.4T 1% /work_backup[root@BAK ~]# vi /etc/fstab#
# /etc/fstab
# Created by anaconda on Mon Aug 29 15:39:38 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=b1724545-924d-4ec1-814b-73301637713a / ext4 defaults 1 1
UUID=B7C5-062C /boot/efi vfat umask=0077,shortname=winnt 0 0
UUID=0a9cb7c1-ee2c-439b-a73e-c55441f9743a /export ext4 defaults 1 2
UUID=98591ed9-25a1-404c-be7f-7740a53f69c7 swap swap defaults 0 0
UUID=583274ec-05f3-49c5-ace7-af9ad144540d /work_backup ext4 defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
Configure iptables:
[root@BAK ~]# vi /etc/sysconfig/iptables# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -s 192.168.1.36/32 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT[root@BAK ~]# iptables-restore < /etc/sysconfig/iptables
Change our ulimits:
[root@BAK ~]# vi /etc/security/limits.conf....
* soft nofile 65535
* hard nofile 65535
* - nofile 65535
Edit our hosts file (host resolution is required by OpenDedup, otherwise a generic Java error will occur). Remember, our IP is 192.168.1.36. Our FQDN is BAK.mynetwork.com:
[root@BAK ~]# vi /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.36 BAK.mynetwork.com
Install OpenDedup and mount our new, deduplicated volume:
[root@BAK ~]# wget http://www.opendedup.org/downloads/sdfs-latest.rpm
[root@BAK ~]# yum -y install jsvc libxml2 java-1.8.0-openjdk fuse
[root@BAK ~]# cd /work_backup/
[root@BAK ~]# mkdir -p /media/pool0
[root@BAK work_backup]# mkfs.sdfs --volume-name=pool0 --volume-capacity=6TB --hash-type=murmur3_128 --io-chunk-size=64 --base-path=/work_backup
[root@BAK work_backup]# mount -t sdfs pool0 /media/pool0/- --volume-name: The name of the volume. THIS IS A REQUIRED OPTION.
- --volume-capacity: Capacity of the volume in [MB|GB|TB]. THIS IS A REQUIRED OPTION
- --hash-type <tiger16|tiger24|murmur3_128>: This is the type of hash engine used to calculate a unique hash. The valid options for hash-type are tiger16 tiger24 murmur3_128 This Defaults to murmur3_128
- --io-chunk-size <SIZE in kB>: The unit size, in kB, of chunks stored. Set this to 4 if you would like to dedup VMDK files inline. Defaults to 128
- --base-path <PATH>: The folder path for all volume data and meta data. Defaults to: /opt/sdfs/<volume name>. Here we wanted our volume to be stored on our external disk so we set that as the base path.
[root@BAK work_backup]# vi /etc/sysctl.conf...
vm.swappiness=1
net.core.rmem_max=254800000
net.core.wmem_max=254800000
net.core.rmem_default=254800000
net.core.wmem_default=254800000
net.core.optmem_max=25480000
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=0
net.core.netdev_max_backlog=250000
net.ipv4.tcp_mem=25480000 25480000 25480000
net.ipv4.tcp_rmem=4096 87380 25480000
net.ipv4.tcp_wmem=4096 65536 25480000
net.ipv4.tcp_low_latency=1[root@BAK work_backup]# sysctl -p
We will not put this volume in fstab, as we need it as an extremely delayed mount, after pretty much every service is finished. So what we can do is either create a service for it, or put it in our crontab. We'll save this one for later. In the meantime, let's check out that Windows share:
[root@BAK work_backup]# smbclient //192.168.0.29/WORK -U "MYNETWORK\Administrator" Enter MYNETWORK\Administrator's password: p@$$W0r(!
Domain=[MYNETWORK] OS=[Windows Server 2003 R2 3790 Service Pack 2] Server=[Windows Server 2003 R2 5.2]smb: \> quit
Yup. Windows Server 2003 SP2. Gotta love this job. Let's automount that share:
[root@BAK work_backup]# cd ~
[root@BAK ~]# vi /etc/auto.master....
/mnt/work /etc/auto.work --timeout=1800 --ghost[root@BAK ~]# mkdir /mnt/work
[root@BAK ~]# vi /etc/auto.workwork_backup -fstype=cifs,rw,noperm,username=MYNETWORK\\Administrator,password=p@$$W0r(!,_netdev,iocharset=utf8 ://192.168.0.29/WORK[root@BAK ~]# chkconfig autofs on
[root@BAK ~]# service autofs startLoading autofs4: [ OK ]
Starting automount: [ OK ][root@BAK ~]# cd /mnt/work/
[root@BAK work]# ls -latotal 4
drwxr-xr-x. 3 root root 0 Aug 29 14:56 .
drwxr-xr-x. 3 root root 4096 Aug 29 14:44 ..
dr-xr-xr-x. 2 root root 0 Aug 29 14:56 work_backup[root@BAK work]# ls -l work_backup/total 1242973
....
It's crucial to:
- Add _netdev as a mount option so as for the system to wait for the network to come up first and then perform the actual filesystem mount
- Add iocharset as a mount option in order to be able to read any special characters our users use, otherwise we'll get a rsync: recv_generator: failed to stat "<filename>" : Invalid or incomplete multibyte or wide character (84) error
- Use a double slash (MYNETWORK\\Administrator) instead of a single one (MYNETWORK\Administrator); since we are in a Linux environment, we need an escape character for this
[root@BAK work]# cd ~
[root@BAK ~]# rsync -avzr /mnt/work/ /media/pool0/ORIGINAL
Here, we just synchronize our share with the directory /media/pool0/ORIGINAL so that in the future, we compare the differences and send them to a third directory (i.e. we'll be creating a sparse backup of just files that have changed from our original backup).
Let's see our savings:
Almost 60%. Yeah, that's pretty decent.
And once that's finished, we add our deduplicated volume mount command and our rsync command and we're all done:
As you can see, we specified the LANG=en_US.UTF-8 environment variable here in order for our environment to be in check with our volume's charset and be safe from the aforementioned rsync: recv_generator: failed to stat "<filename>" : Invalid or incomplete multibyte or wide character (84) error.
If you ever need to delete a volume, you need to unmount it first and then remove the data, log and configuration files. My deduplicated volume is mounted on /media/pool0, the volume name is pool0 and my actual disk is mounted on /work_backup:
Let's see our savings:
[root@BAK ~]# sdfscli --volume-infoVolume Capacity : 6 TB
Volume Current Logical Size : 567.57 GB
Volume Max Percentage Full : 95.0%
Volume Duplicate Data Written : 395.33 GB
Unique Blocks Stored: 227.61 GB
Unique Blocks Stored after Compression : 227.7 GB
Cluster Block Copies : 2
Volume Virtual Dedup Rate (Unique Blocks Stored/Current Size) : -4.56%
Volume Actual Storage Savings (Compressed Unique Blocks Stored/Current Size) : 59.88%
Compression Rate: -0.04%
Almost 60%. Yeah, that's pretty decent.
And once that's finished, we add our deduplicated volume mount command and our rsync command and we're all done:
[root@BAK work]# cd ~
[root@BAK ~]# crontab -e@reboot /bin/sleep 60;/bin/mount -t sdfs pool0 /media/pool0/ #Mount our deduplicated volume a minute after booting
#15 17 * * * LANG=en_US.UTF-8;append=`/bin/date +"\%d\%m\%Y"`;if /usr/bin/test `/usr/bin/pgrep rsync | wc -l` -lt 1 ; then /bin/mkdir /media/pool0/$append;/usr/bin/rsync -avzr --log-file=/root/work_backup_$append.log --compare-dest=/media/pool0/ORIGINAL /mnt/work/ /media/pool0/$append; fi #We can scan if other rsync processes are still running and not start, it can apply in some environments
15 17 * * * LANG=en_US.UTF-8;append=`/bin/date +"\%d\%m\%Y"`;/bin/mkdir /media/pool0/$append;/usr/bin/rsync -avzr --log-file=/root/work_backup_$append.log --compare-dest=/media/pool0/ORIGINAL /mnt/work/ /media/pool0/$append #Check the differences between our Windows Share and our initial backup directory and put any differing files in /media/pool0/DayMonthYear
00 05 * * * LANG=en_US.UTF-8;append=`/bin/date +"\%d\%m\%Y" --date="yesterday"`;/usr/bin/rsync -av --ignore-existing /media/pool0/$append/work_backup/ /media/pool0/ORIGINAL/work_backup/ #If we want, we can actually put any new files in the "ORIGINAL" directory so that they don't get copied every time in order to save bandwidth and time. Any pre-existing files will not be overwritten.
As you can see, we specified the LANG=en_US.UTF-8 environment variable here in order for our environment to be in check with our volume's charset and be safe from the aforementioned rsync: recv_generator: failed to stat "<filename>" : Invalid or incomplete multibyte or wide character (84) error.
If you ever need to delete a volume, you need to unmount it first and then remove the data, log and configuration files. My deduplicated volume is mounted on /media/pool0, the volume name is pool0 and my actual disk is mounted on /work_backup:
[root@BAK ~]# umount /media/pool0/
[root@BAK ~]# rm -rf /work_backup/*
[root@BAK ~]# rm -f /var/log/sdfs/pool0-volume-cfg.xml*
[root@BAK ~]# rm -f /etc/sdfs/pool0-volume-cfg.xml
Creating a backup server on Linux is a pretty complicated task. Thank you for sharing a detailed and accessible guide!
ReplyDelete