Wednesday, July 9, 2014

Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 2

All right. Time to get serious with this.
root@zen-lb:~# apt-get update
root@zen-lb:~# apt-get upgrade

We 'll need these tools later on:
root@zen-lb:~# apt-get install build-essential devscripts m4 quilt debhelper zlib1g-dev bc gcc++ cmake

If PCRE, tcmalloc (from the Google perftools package) and/or Hoard are available Pound will link against them. This will provide a significant performance boost and is highly recommended. Guess what:
root@zen-lb:~# apt-get install libpcrecpp0 libpcre3-dev libpcre3 libpcre++0 libpcre++-dev libtcmalloc-minimal4 libgoogle-perftools4 libgoogle-perftools-dev
root@zen-lb:~# mkdir hoard
root@zen-lb:~# cd hoard/
root@zen-lb:~/hoard# wget https://github.com/emeryberger/Hoard/releases/download/3.10/Hoard-3.10-source.tar.gz
root@zen-lb:~/hoard# gunzip Hoard-3.10-source.tar.gz 
root@zen-lb:~/hoard# tar -xf Hoard-3.10-source.tar 
root@zen-lb:~/hoard# cd Hoard/src
root@zen-lb:~/hoard/Hoard/src# make linux-gcc-x86
root@zen-lb:~/hoard/Hoard/src# cp libhoard.so /usr/lib/.

Add this to our /etc/profile so that the hoard library is loaded:
root@zen-lb:~/hoard/Hoard/src# vi /etc/profile
export LD_PRELOAD=/usr/lib/libhoard.so

Test that it's loaded:
root@zen-lb:~# ldd /bin/ls
        linux-gate.so.1 =>  (0xb77cd000)
        /usr/lib/libhoard.so (0xb7789000)
        libselinux.so.1 => /lib/i386-linux-gnu/libselinux.so.1 (0xb7762000)
        librt.so.1 => /lib/i386-linux-gnu/i686/cmov/librt.so.1 (0xb7758000)
        libacl.so.1 => /lib/i386-linux-gnu/libacl.so.1 (0xb774e000)
        libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb75ea000)
        libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb75e6000)
        libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb75cd000)
        libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb74e0000)
        libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xb74ba000)
        libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb749d000)
        /lib/ld-linux.so.2 (0xb77ce000)
        libattr.so.1 => /lib/i386-linux-gnu/libattr.so.1 (0xb7497000)

Great. Time to increase our SSL security. A few notes:

To increase SSL security even further, we need to upgrade SSL and make it deny any compression.

To mitigate CRIME we need to disable ssl-compression
To mitigate BEAST and Lucky13 we need to use TLS 1.1 and above, no SSLv3 and no RC4
To mitigate BREACH we need to disable http-compression (gzip)

Also, to support strict transport security you need to add this to nginx.conf:

add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";

And httpd.conf:

LoadModule headers_module modules/mod_headers.so

<VirtualHost 192.168.99.99:80>
    Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"
</VirtualHost>


All right, let's get to configuring our OpenSSL:
root@zen-lb:~/hoard/Hoard/src# cd ~
root@zen-lb:~# mkdir openssl
root@zen-lb:~# cd openssl
root@zen-lb:~/openssl# apt-get source openssl
root@zen-lb:~/openssl# cd openssl-*
root@zen-lb:~/openssl/openssl-1.0.1e# quilt pop -a

Now, we need to disable compression, insecure ciphers, sslv2 (and sslv3):
root@zen-lb:~/openssl/openssl-1.0.1e# vi debian/rules
   CONFARGS  = -no-comp --prefix=/usr --openssldir=/usr/lib/ssl --libdir=lib/$(DEB_HOST_MULTIARCH) no-idea no-mdc2 no-rc5 no-zlib  enable-tlsext no-ssl2   #Include no-ssl3 for even better security.

Commit the changes:
root@zen-lb:~/openssl/openssl-1.0.1e# quilt push -a
root@zen-lb:~/openssl/openssl-1.0.1e# dpkg-source --commit
root@zen-lb:~/openssl/openssl-1.0.1e# debuild -uc -us
root@zen-lb:~/openssl/openssl-1.0.1e# cd ..
root@zen-lb:~/openssl# dpkg -i *ssl*.deb

Let's mark our openssl-related packages as non-upgradable. We do this so they don't get upgraded automatically to the vanilla one. This obviously means that we'll have to upgrade OpenSSL manually every time, just as we did earlier.
root@zen-lb:~/openssl# apt-mark hold libssl-dev libssl-doc libssl openssl libssl1.0.0 libssl1.0.0-dbg
root@zen-lb:~/openssl# reboot

Now, we need to upgrade Pound. Zen uses Pound for HTTP and HTTPS farms. Unfortunately, its version is old and does not support more modern SSL methods that are essential for SSL security. You need at least version 2.6f, which is very stable.

If you want the latest, check out if there are any bugs that may disrupt your service first. If you need elliptic curve cryptography support, I recommend the customized code by Joe Gooch, which can be found here. You need to make sure you select the "stage for upstream" branch. Joe Gooch's versions have some extra features as well, such as "DisableSSLv3","DisableTLSv10", "DisableTLSv11" and "DisableTLSv12".

In this example, I'm going to go ahead and use Pound 2.7c.
root@zen-lb:~# mkdir pound
root@zen-lb:~# cd pound
root@zen-lb:~/pound# wget https://fossies.org/linux/www/Pound-2.7c.tar.bz2
root@zen-lb:~/pound# bunzip2 Pound-2.7c.tar.bz2
root@zen-lb:~/pound# tar -xf Pound-2.7c.tar
root@zen-lb:~/pound# cd Pound-2.7c
root@zen-lb:~/pound/Pound-2.7c# ./configure
root@zen-lb:~/pound/Pound-2.7c# make
root@zen-lb:~/pound/Pound-2.7c# cp pound /usr/local/zenloadbalancer/app/pound/sbin/pound2.7c
root@zen-lb:~/pound/Pound-2.7c# cp poundctl /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.7c
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/pound /usr/local/zenloadbalancer/app/pound/sbin/pound2.5
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/poundctl /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.5
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/pound2.7c /usr/local/zenloadbalancer/app/pound/sbin/pound
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.7c /usr/local/zenloadbalancer/app/pound/sbin/poundctl
root@zen-lb:~/pound/Pound-2.7c# cd ~

Let's see if our pound is linked to the extra libraries we installed:
root@zen-lb:~# ldd /usr/local/zenloadbalancer/app/pound/sbin/pound
        linux-gate.so.1 =>  (0xb7751000)
        /usr/lib/libhoard.so (0xb770d000)
        libpcreposix.so.3 => /usr/lib/i386-linux-gnu/libpcreposix.so.3 (0xb7703000)
        libssl.so.1.0.0 => /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0 (0xb76aa000)
        libcrypto.so.1.0.0 => /usr/lib/i386-linux-gnu/i686/cmov/libcrypto.so.1.0.0 (0xb74ed000)
        libresolv.so.2 => /lib/i386-linux-gnu/i686/cmov/libresolv.so.2 (0xb74d9000)
        libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb74d5000)
        libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xb74af000)
        libtcmalloc.so.4 => /usr/lib/libtcmalloc.so.4 (0xb7447000)
        libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb742e000)
        libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb72ca000)
        libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb71de000)
        libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb71c1000)
        libpcre.so.3 => /lib/i386-linux-gnu/libpcre.so.3 (0xb7182000)
        /lib/ld-linux.so.2 (0xb7752000)

Great. Now, let's create a farm and test it out:


Note that if you're trying to find good ciphers to work with, Mozilla have some great recommendations here. There's also a great list here.

Well, everything looks good, so what about our new options? Let's manually edit our farmname_pound.cfg file:
root@zen-lb:~# vi /usr/local/zenloadbalancer/config/HTTPSTEST_pound.cfg
...
Threads  512
...
Cert "/usr/local/zenloadbalancer/config/zencert.pem"
        SSLHonorCipherOrder     1
        SSLAllowClientRenegotiation     0


The "Threads" variable exists in pound 2.7 and above so be careful about using it if you've downloaded a version in the 2.6 tree.

The SSLHonorCipherOrder 1 variable indicates that the order of the encryption processes that is defined in your Ciphers list must be respected by the web server, therefore the first cipher match found must be used.

The SSLAllowClientRenegotiation 0 variable indicates that no client renegotiation will be honored. When set to 0, no client renegotiation will be honored.  When 1, secure renegotiation will be honored.  When 2, insecure renegotiation will be honored.

Now, go and restart your farm from the UI. Test that everything works:

Great. Just in case, let's test that our SSL farm does not support insecure renegotiation (assuming my load balancer's external IP is 192.168.0.30):

root@zen-lb:~# openssl s_client -connect 192.168.0.30:443

and press "R" and return to renegotiate and then do "GET / HTTP /1.0". If you get a response, you've done something wrong.


 And that concludes part 2.

Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 1

I'm a bit partial to Zen Load Balancer. As a matter of a fact, I love it. It has many, many things ready to go from the start.

And it being just a Debian distro with the zenloadbalancer package on top, there's a lot you can do to customize it. The first thing we need to do is get rid of this:



My system has 16GB of memory but memory reported is just 3GB? Yup.
See Zen Load Balancer is a 32-bit app and it is distributed with a 32-bit Debian distro.

Assuming we have a 64-bit system with more memory installed we'll need to upgrade the kernel to a PAE one. This is both a performance and a security enhancement. It will allow us to use more memory and will also enable NX protection (provided that our BIOS and CPU support it too), as the NX bit works on the 63rd bit of the address.

Editing our repos first:
root@zen-lb:~# vi /etc/apt/sources.list
#official repository for Debian
deb http://ftp.debian.org/debian/ stable main non-free
deb-src http://ftp.debian.org/debian/ stable main non-free
deb http://security.debian.org/ stable/updates main
deb-src http://security.debian.org/ stable/updates main
#official repository for Zen Load Balancer Updates
deb http://zenloadbalancer.sourceforge.net/apt/x86 v3/

#Let's add this repo as well to do a moderate PAE upgrade at first
deb http://security.debian.org/debian-security squeeze/updates main

Let's try to upgrade our kernel now:
root@zen-lb:~# apt-get update
....
....
....
Reading package lists... Done
W: GPG error: http://ftp.debian.org stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B48AD6246925553 NO_PUBKEY 6FB2A1C265FFB764
W: GPG error: http://security.debian.org stable/updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B48AD6246925553

Yeah ok, classic Debian thing:
root@zen-lb:~# gpg --keyserver pgpkeys.mit.edu --recv-key 6FB2A1C265FFB764
root@zen-lb:~# gpg -a --export 6FB2A1C265FFB764 | apt-key add -
root@zen-lb:~# gpg --keyserver pgpkeys.mit.edu --recv-key 8B48AD6246925553
root@zen-lb:~# gpg -a --export 8B48AD6246925553 | apt-key add -
root@zen-lb:~# apt-get update
Fetched 548 kB in 1s (444 kB/s)
Reading package lists... Done

One more time for the world:
root@zen-lb:~# apt-cache search linux-image
linux-image-2.6-486 - Linux for older PCs (dummy package)
linux-image-2.6-686 - Linux for modern PCs (dummy package)
linux-image-2.6-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
linux-image-2.6-686-pae - Linux for modern PCs (dummy package)
linux-image-2.6-amd64 - Linux for 64-bit PCs (dummy package)
linux-image-486 - Linux for older PCs (meta-package)
linux-image-686 - Linux for modern PCs (dummy package)
linux-image-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
linux-image-686-pae - Linux for modern PCs (meta-package)
linux-image-amd64 - Linux for 64-bit PCs (meta-package)
linux-image-rt-686-pae - Linux for modern PCs (meta-package), PREEMPT_RT
linux-image-3.2.0-4-486 - Linux 3.2 for older PCs
linux-image-3.2.0-4-686-pae - Linux 3.2 for modern PCs
linux-image-3.2.0-4-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-686-pae
linux-image-3.2.0-4-amd64 - Linux 3.2 for 64-bit PCs
linux-image-3.2.0-4-rt-686-pae - Linux 3.2 for modern PCs, PREEMPT_RT
linux-image-3.2.0-4-rt-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-rt-686-pae
linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486
linux-headers-2.6.32-5-686 - Header files for Linux 2.6.32-5-686
linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem
linux-headers-2.6.32-5-amd64 - Header files for Linux 2.6.32-5-amd64
linux-headers-2.6.32-5-openvz-686 - Header files for Linux 2.6.32-5-openvz-686
linux-headers-2.6.32-5-vserver-686 - Header files for Linux 2.6.32-5-vserver-686
linux-headers-2.6.32-5-vserver-686-bigmem - Header files for Linux 2.6.32-5-vserver-686-bigmem
linux-headers-2.6.32-5-xen-686 - Header files for Linux 2.6.32-5-xen-686
linux-image-2.6.32-5-486 - Linux 2.6.32 for old PCs
linux-image-2.6.32-5-686 - Linux 2.6.32 for modern PCs
linux-image-2.6.32-5-686-bigmem - Linux 2.6.32 for PCs with 4GB+ RAM
linux-image-2.6.32-5-686-bigmem-dbg - Debugging infos for Linux 2.6.32-5-686-bigmem
linux-image-2.6.32-5-amd64 - Linux 2.6.32 for 64-bit PCs
linux-image-2.6.32-5-openvz-686 - Linux 2.6.32 for modern PCs, OpenVZ support
linux-image-2.6.32-5-openvz-686-dbg - Debugging infos for Linux 2.6.32-5-openvz-686
linux-image-2.6.32-5-vserver-686 - Linux 2.6.32 for modern PCs, Linux-VServer support
linux-image-2.6.32-5-vserver-686-bigmem - Linux 2.6.32 for PCs with 4GB+ RAM, Linux-VServer support
linux-image-2.6.32-5-vserver-686-bigmem-dbg - Debugging infos for Linux 2.6.32-5-vserver-686-bigmem
linux-image-2.6.32-5-xen-686 - Linux 2.6.32 for modern PCs, Xen dom0 support
linux-image-2.6.32-5-xen-686-dbg - Debugging infos for Linux 2.6.32-5-xen-686

Right, let's be conservative and upgrade to a 2.6 PAE kernel, we'll do a major upgrade later:
root@zen-lb:~# uname -a
Linux zen-lb 2.6.32-5-686 #1 SMP Wed Jan 12 04:01:41 UTC 2011 i686 GNU/Linux
root@zen-lb:~# apt-get install linux-image-2.6.32-5-686-bigmem
Get:1 http://ftp.debian.org/debian/ stable/main linux-base all 3.5 [34.3 kB]
Get:2 http://security.debian.org/debian-security/ squeeze/updates/main linux-image-2.6.32-5-686-bigmem i386 2.6.32-48squeeze6 [27.6 MB]
Get:3 http://ftp.debian.org/debian/ stable/main firmware-linux-free all 3.2 [20.7 kB]
Fetched 27.7 MB in 20s (1,366 kB/s)
Preconfiguring packages ...
(Reading database ... 18065 files and directories currently installed.)
Preparing to replace linux-base 2.6.32-30 (using .../linux-base_3.5_all.deb) ...
Unpacking replacement linux-base ...
Selecting previously deselected package linux-image-2.6.32-5-686-bigmem.
Unpacking linux-image-2.6.32-5-686-bigmem (from .../linux-image-2.6.32-5-686-bigmem_2.6.32-48squeeze6_i386.deb) ...
Selecting previously deselected package firmware-linux-free.
Unpacking firmware-linux-free (from .../firmware-linux-free_3.2_all.deb) ...
Processing triggers for man-db ...
Setting up linux-base (3.5) ...
Setting up linux-image-2.6.32-5-686-bigmem (2.6.32-48squeeze6) ...
Running depmod.
Running update-initramfs.
update-initramfs: Generating /boot/initrd.img-2.6.32-5-686-bigmem
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 2.6.32-5-686-bigmem /boot/vmlinuz-2.6.32-5-686-bigmem
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 2.6.32-5-686-bigmem /boot/vmlinuz-2.6.32-5-686-bigmem
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-5-686-bigmem
Found initrd image: /boot/initrd.img-2.6.32-5-686-bigmem
Found linux image: /boot/vmlinuz-2.6.32-5-686
Found initrd image: /boot/initrd.img-2.6.32-5-686
done
Setting up firmware-linux-free (3.2) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-2.6.32-5-686-bigmem
root@zen-lb:~# reboot

All right, so did it work?

Looks good. And what about NX?
root@zen-lb:~# dmesg | grep ".*NX.*protection"
[    0.000000] NX (Execute Disable) protection: active

Cool. Let's go on then. Let's do a distro upgrade from Squeeze to Wheezy:
root@zen-lb:~# apt-get dist-upgrade
root@zen-lb:~# reboot

Indeed:
root@zen-lb:~# cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"
NAME="Debian GNU/Linux"
VERSION_ID="7"
VERSION="7 (wheezy)"
ID=debian
ANSI_COLOR="1;31"
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support/"
BUG_REPORT_URL="http://bugs.debian.org/"
root@zen-lb:~# cat /proc/version
Linux version 3.2.0-4-686-pae (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.60-1+deb7u1

The first thing I'm going to do is tune my filesystem. To be honest, this is just a load balancer so I can afford to lose a few seconds of logs if the power goes down.
root@zen-lb:~# vi /etc/fstab
proc            /proc           proc    defaults        0       0
# / was on /dev/sdb3 during installation
UUID=b6016824-536e-43bc-8f1f-fbfd2fab146d /               ext4    noatime,nodiratime,nobarrier,nobh,commit=120,data=writeback,journal_async_commit,errors=remount-ro 0       1
# /boot was on /dev/sdb1 during installation
UUID=6d4bd9ca-ba29-4700-b90c-07c614d79f0e /boot           ext4    defaults        0       2
# swap was on /dev/sdb2 during installation
UUID=3cb4eb4d-b0ab-4b60-825b-fc0224356580 none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

Go to single user mode and tune my root filesystem (mine is on /dev/sdb3):
root@zen-lb:~# init 1
root@zen-lb:~# tune2fs -O dir_index /dev/sdb3
root@zen-lb:~# umount -a
root@zen-lb:~# e2fsck -D /dev/sdb3

And now increase the number of open files limit:
root@zen-lb:~# vi /etc/security/limits.conf
....
* soft nofile 65536
* hard nofile 65536
# End of file

When regular users log in, they get an open files warning, let's correct it by uncommenting these few lines of code in /etc/profile:
root@zen-lb:~# vi /etc/profile
....
#if [ -f /etc/sysctl.conf ]; then
# FILEMAX=`grep "^fs.file-max.*=" /etc/sysctl.conf | awk -F'=' '{printf $2}'`
# if [ "$FILEMAX" != "" ]; then
#  ulimit -n $FILEMAX
# fi
#fi

Finally, let's update our repos to correctly receive wheezy updates:
root@zen-lb:~# vi /etc/apt/sources.list
#official repository for Debian
deb http://ftp.debian.org/debian wheezy main contrib non-free
deb-src http://ftp.debian.org/debian wheezy main contrib non-free
deb http://ftp.debian.org/debian wheezy-updates main contrib non-free

deb http://http.debian.net/debian wheezy main contrib non-free
deb-src http://http.debian.net/debian wheezy main contrib non-free

deb http://http.debian.net/debian wheezy-updates main contrib non-free
deb-src http://http.debian.net/debian wheezy-updates main contrib non-free

deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free

#official repository for Zen Load Balancer Updates
deb http://zenloadbalancer.sourceforge.net/apt/x86 v3/

All right, I guess we didn't do that much, but it's enough to call the end of Part 1.