The Sysadminosaurus' IT blog: Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 1

I'm a bit partial to Zen Load Balancer. As a matter of a fact, I love it. It has many, many things ready to go from the start.

And it being just a Debian distro with the zenloadbalancer package on top, there's a lot you can do to customize it. The first thing we need to do is get rid of this:

My system has 16GB of memory but memory reported is just 3GB? Yup.
See Zen Load Balancer is a 32-bit app and it is distributed with a 32-bit Debian distro.

Assuming we have a 64-bit system with more memory installed we'll need to upgrade the kernel to a PAE one. This is both a performance and a security enhancement. It will allow us to use more memory and will also enable NX protection (provided that our BIOS and CPU support it too), as the NX bit works on the 63rd bit of the address.

Editing our repos first:

root@zen-lb:~# vi /etc/apt/sources.list#official repository for Debian
deb http://ftp.debian.org/debian/ stable main non-free
deb-src http://ftp.debian.org/debian/ stable main non-free
deb http://security.debian.org/ stable/updates main
deb-src http://security.debian.org/ stable/updates main
#official repository for Zen Load Balancer Updates
deb http://zenloadbalancer.sourceforge.net/apt/x86 v3/

#Let's add this repo as well to do a moderate PAE upgrade at first
deb http://security.debian.org/debian-security squeeze/updates main

Let's try to upgrade our kernel now:

root@zen-lb:~# apt-get update....
....
....
Reading package lists... Done
W: GPG error: http://ftp.debian.org stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B48AD6246925553 NO_PUBKEY 6FB2A1C265FFB764
W: GPG error: http://security.debian.org stable/updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B48AD6246925553

Yeah ok, classic Debian thing:

root@zen-lb:~# gpg --keyserver pgpkeys.mit.edu --recv-key 6FB2A1C265FFB764
root@zen-lb:~# gpg -a --export 6FB2A1C265FFB764 | apt-key add -
root@zen-lb:~# gpg --keyserver pgpkeys.mit.edu --recv-key 8B48AD6246925553
root@zen-lb:~# gpg -a --export 8B48AD6246925553 | apt-key add -
root@zen-lb:~# apt-get updateFetched 548 kB in 1s (444 kB/s)
Reading package lists... Done

One more time for the world:

root@zen-lb:~# apt-cache search linux-imagelinux-image-2.6-486 - Linux for older PCs (dummy package)
linux-image-2.6-686 - Linux for modern PCs (dummy package)
linux-image-2.6-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
linux-image-2.6-686-pae - Linux for modern PCs (dummy package)
linux-image-2.6-amd64 - Linux for 64-bit PCs (dummy package)
linux-image-486 - Linux for older PCs (meta-package)
linux-image-686 - Linux for modern PCs (dummy package)
linux-image-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
linux-image-686-pae - Linux for modern PCs (meta-package)
linux-image-amd64 - Linux for 64-bit PCs (meta-package)
linux-image-rt-686-pae - Linux for modern PCs (meta-package), PREEMPT_RT
linux-image-3.2.0-4-486 - Linux 3.2 for older PCs
linux-image-3.2.0-4-686-pae - Linux 3.2 for modern PCs
linux-image-3.2.0-4-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-686-pae
linux-image-3.2.0-4-amd64 - Linux 3.2 for 64-bit PCs
linux-image-3.2.0-4-rt-686-pae - Linux 3.2 for modern PCs, PREEMPT_RT
linux-image-3.2.0-4-rt-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-rt-686-pae
linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486
linux-headers-2.6.32-5-686 - Header files for Linux 2.6.32-5-686
linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem
linux-headers-2.6.32-5-amd64 - Header files for Linux 2.6.32-5-amd64
linux-headers-2.6.32-5-openvz-686 - Header files for Linux 2.6.32-5-openvz-686
linux-headers-2.6.32-5-vserver-686 - Header files for Linux 2.6.32-5-vserver-686
linux-headers-2.6.32-5-vserver-686-bigmem - Header files for Linux 2.6.32-5-vserver-686-bigmem
linux-headers-2.6.32-5-xen-686 - Header files for Linux 2.6.32-5-xen-686
linux-image-2.6.32-5-486 - Linux 2.6.32 for old PCs
linux-image-2.6.32-5-686 - Linux 2.6.32 for modern PCs
linux-image-2.6.32-5-686-bigmem - Linux 2.6.32 for PCs with 4GB+ RAM
linux-image-2.6.32-5-686-bigmem-dbg - Debugging infos for Linux 2.6.32-5-686-bigmem
linux-image-2.6.32-5-amd64 - Linux 2.6.32 for 64-bit PCs
linux-image-2.6.32-5-openvz-686 - Linux 2.6.32 for modern PCs, OpenVZ support
linux-image-2.6.32-5-openvz-686-dbg - Debugging infos for Linux 2.6.32-5-openvz-686
linux-image-2.6.32-5-vserver-686 - Linux 2.6.32 for modern PCs, Linux-VServer support
linux-image-2.6.32-5-vserver-686-bigmem - Linux 2.6.32 for PCs with 4GB+ RAM, Linux-VServer support
linux-image-2.6.32-5-vserver-686-bigmem-dbg - Debugging infos for Linux 2.6.32-5-vserver-686-bigmem
linux-image-2.6.32-5-xen-686 - Linux 2.6.32 for modern PCs, Xen dom0 support
linux-image-2.6.32-5-xen-686-dbg - Debugging infos for Linux 2.6.32-5-xen-686

Right, let's be conservative and upgrade to a 2.6 PAE kernel, we'll do a major upgrade later:

root@zen-lb:~# uname -aLinux zen-lb 2.6.32-5-686 #1 SMP Wed Jan 12 04:01:41 UTC 2011 i686 GNU/Linuxroot@zen-lb:~# apt-get install linux-image-2.6.32-5-686-bigmemGet:1 http://ftp.debian.org/debian/ stable/main linux-base all 3.5 [34.3 kB]
Get:2 http://security.debian.org/debian-security/ squeeze/updates/main linux-image-2.6.32-5-686-bigmem i386 2.6.32-48squeeze6 [27.6 MB]
Get:3 http://ftp.debian.org/debian/ stable/main firmware-linux-free all 3.2 [20.7 kB]
Fetched 27.7 MB in 20s (1,366 kB/s)
Preconfiguring packages ...
(Reading database ... 18065 files and directories currently installed.)
Preparing to replace linux-base 2.6.32-30 (using .../linux-base_3.5_all.deb) ...
Unpacking replacement linux-base ...
Selecting previously deselected package linux-image-2.6.32-5-686-bigmem.
Unpacking linux-image-2.6.32-5-686-bigmem (from .../linux-image-2.6.32-5-686-bigmem_2.6.32-48squeeze6_i386.deb) ...
Selecting previously deselected package firmware-linux-free.
Unpacking firmware-linux-free (from .../firmware-linux-free_3.2_all.deb) ...
Processing triggers for man-db ...
Setting up linux-base (3.5) ...
Setting up linux-image-2.6.32-5-686-bigmem (2.6.32-48squeeze6) ...
Running depmod.
Running update-initramfs.
update-initramfs: Generating /boot/initrd.img-2.6.32-5-686-bigmem
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 2.6.32-5-686-bigmem /boot/vmlinuz-2.6.32-5-686-bigmem
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 2.6.32-5-686-bigmem /boot/vmlinuz-2.6.32-5-686-bigmem
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-5-686-bigmem
Found initrd image: /boot/initrd.img-2.6.32-5-686-bigmem
Found linux image: /boot/vmlinuz-2.6.32-5-686
Found initrd image: /boot/initrd.img-2.6.32-5-686
done
Setting up firmware-linux-free (3.2) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-2.6.32-5-686-bigmemroot@zen-lb:~# reboot

All right, so did it work?

Looks good. And what about NX?

root@zen-lb:~# dmesg | grep ".*NX.*protection"[    0.000000] NX (Execute Disable) protection: active

Cool. Let's go on then. Let's do a distro upgrade from Squeeze to Wheezy:

root@zen-lb:~# apt-get dist-upgrade
root@zen-lb:~# reboot

Indeed:

root@zen-lb:~# cat /etc/*releasePRETTY_NAME="Debian GNU/Linux 7 (wheezy)"
NAME="Debian GNU/Linux"
VERSION_ID="7"
VERSION="7 (wheezy)"
ID=debian
ANSI_COLOR="1;31"
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support/"
BUG_REPORT_URL="http://bugs.debian.org/"root@zen-lb:~# cat /proc/versionLinux version 3.2.0-4-686-pae (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.60-1+deb7u1

The first thing I'm going to do is tune my filesystem. To be honest, this is just a load balancer so I can afford to lose a few seconds of logs if the power goes down.

root@zen-lb:~# vi /etc/fstabproc            /proc           proc    defaults        0       0
# / was on /dev/sdb3 during installation
UUID=b6016824-536e-43bc-8f1f-fbfd2fab146d /               ext4    noatime,nodiratime,nobarrier,nobh,commit=120,data=writeback,journal_async_commit,errors=remount-ro 0       1
# /boot was on /dev/sdb1 during installation
UUID=6d4bd9ca-ba29-4700-b90c-07c614d79f0e /boot           ext4    defaults        0       2
# swap was on /dev/sdb2 during installation
UUID=3cb4eb4d-b0ab-4b60-825b-fc0224356580 none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

Go to single user mode and tune my root filesystem (mine is on /dev/sdb3):

root@zen-lb:~# init 1
root@zen-lb:~# tune2fs -O dir_index /dev/sdb3
root@zen-lb:~# umount -a
root@zen-lb:~# e2fsck -D /dev/sdb3

And now increase the number of open files limit:

root@zen-lb:~# vi /etc/security/limits.conf....
* soft nofile 65536
* hard nofile 65536
# End of file

When regular users log in, they get an open files warning, let's correct it by uncommenting these few lines of code in /etc/profile:

root@zen-lb:~# vi /etc/profile....
#if [ -f /etc/sysctl.conf ]; then
# FILEMAX=`grep "^fs.file-max.*=" /etc/sysctl.conf | awk -F'=' '{printf $2}'`
# if [ "$FILEMAX" != "" ]; then
#  ulimit -n $FILEMAX
# fi
#fi

Finally, let's update our repos to correctly receive wheezy updates:

root@zen-lb:~# vi /etc/apt/sources.list#official repository for Debian
deb http://ftp.debian.org/debian wheezy main contrib non-free
deb-src http://ftp.debian.org/debian wheezy main contrib non-free
deb http://ftp.debian.org/debian wheezy-updates main contrib non-free

deb http://http.debian.net/debian wheezy main contrib non-free
deb-src http://http.debian.net/debian wheezy main contrib non-free

deb http://http.debian.net/debian wheezy-updates main contrib non-free
deb-src http://http.debian.net/debian wheezy-updates main contrib non-free

deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free

#official repository for Zen Load Balancer Updates
deb http://zenloadbalancer.sourceforge.net/apt/x86 v3/

All right, I guess we didn't do that much, but it's enough to call the end of Part 1.

8 comments:

AnonymousMay 28, 2015 at 6:49 PM
Hi nik,

I was wondering if you could help me out. I have followed your instructions to a T and it breaks my ZLB install every time. I am running ZLB in a xenserver 6.5 environment and if I just do a plain install using the v3.05 iso everything works. I am loadbalancing using the LX4NAT profile with DNAT and it works. As soon as I do the above or upgrade the kernel to a xen friendly version so that I can install xen-tools, ZLB no longer forwards the packets to the farm. It still hosts the VIP, does HA, configures farms and interfaces, but no routing is done.

I'd like to be able to do both - upgrade the environment to allow for greater memory usage and be xen aware. I suspect it's something simple (kernel config) but I can't work out where it is going wrong or resolve this issue - not through lack of trying.

Cheers

Chris
UnknownMay 28, 2015 at 10:31 PM
Let me do a few tests on a XenServer 6.5 and I'll let you know. Should be over the weekend. But the first thing that springs to mind is that Zen on Xen should be installed using a normal Debian distro ISO and then adding the Zen LB repo and using apt-get install. Anyway, I'll let you know.
AnonymousMay 29, 2015 at 7:23 PM
Hi Nik,

I got it sorted. It was a combination of a few things and it was driving me insane. So number one was I had one of the interfaces bonded with LACP on the XenServer side and it wasn't playing nice with the switch for a start. Then there were some iptables issues with my routing config which weren't a problem in the base iso but appeared after I did the kernel updates.

Your instructions were awesome though and thanks for the reply. I might reach out again if I have any further issues.

Cheers

Chris
UnknownJuly 8, 2015 at 4:04 PM
Hi Nik,
Since you seem to be pretty well in touch with how the inner workings of zlb works, have you ever tried to set one up on a vanilla Debian install? I know it should just be a matter of adding the repository, but it doesn't seem to want to read anything when running apt-get update, so nothing beyond that works. The reason I ask, is that I would like to try this out as a linux container (lxc) to help out with a small HA install I'm helping with and having only a few servers available, zlb as a container would be very useful.

I'll try a Debian install again tonight, maybe the Sourceforge repo was just down for some reason.

Thanks!
Brad
UnknownJuly 13, 2015 at 6:49 AM
Hi Nik,
I found the issue. ZLB is 32 bit, and I was starting with a 64 Bit stripped down Squeeze install. It would parse the repo, but wasn't able to install it. I made a 32bit Debian squeeze install and did the same steps and it loaded up without any issue. I'll try again with a 64 install and see if adding the 32bit compatibility pieces will allow it to run on a 64 bit machine.
Didn't work: lxc-create -t download -n zlb1 -- --dist debian --release squeeze --arch amd64
Worked: lxc-create -t download -n zlb1 -- --dist debian --release squeeze --arch i386
So, I'm on my way regardless. Thanks for the tips, I'm off to try a newer version and 64 bit.
Thanks!
Brad

Wednesday, July 9, 2014

Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 1

8 comments: