Wednesday, July 9, 2014

Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 2

All right. Time to get serious with this.
root@zen-lb:~# apt-get update
root@zen-lb:~# apt-get upgrade

We 'll need these tools later on:
root@zen-lb:~# apt-get install build-essential devscripts m4 quilt debhelper zlib1g-dev bc gcc++ cmake

If PCRE, tcmalloc (from the Google perftools package) and/or Hoard are available Pound will link against them. This will provide a significant performance boost and is highly recommended. Guess what:
root@zen-lb:~# apt-get install libpcrecpp0 libpcre3-dev libpcre3 libpcre++0 libpcre++-dev libtcmalloc-minimal4 libgoogle-perftools4 libgoogle-perftools-dev
root@zen-lb:~# mkdir hoard
root@zen-lb:~# cd hoard/
root@zen-lb:~/hoard# wget https://github.com/emeryberger/Hoard/releases/download/3.10/Hoard-3.10-source.tar.gz
root@zen-lb:~/hoard# gunzip Hoard-3.10-source.tar.gz 
root@zen-lb:~/hoard# tar -xf Hoard-3.10-source.tar 
root@zen-lb:~/hoard# cd Hoard/src
root@zen-lb:~/hoard/Hoard/src# make linux-gcc-x86
root@zen-lb:~/hoard/Hoard/src# cp libhoard.so /usr/lib/.

Add this to our /etc/profile so that the hoard library is loaded:
root@zen-lb:~/hoard/Hoard/src# vi /etc/profile
export LD_PRELOAD=/usr/lib/libhoard.so

Test that it's loaded:
root@zen-lb:~# ldd /bin/ls
        linux-gate.so.1 =>  (0xb77cd000)
        /usr/lib/libhoard.so (0xb7789000)
        libselinux.so.1 => /lib/i386-linux-gnu/libselinux.so.1 (0xb7762000)
        librt.so.1 => /lib/i386-linux-gnu/i686/cmov/librt.so.1 (0xb7758000)
        libacl.so.1 => /lib/i386-linux-gnu/libacl.so.1 (0xb774e000)
        libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb75ea000)
        libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb75e6000)
        libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb75cd000)
        libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb74e0000)
        libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xb74ba000)
        libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb749d000)
        /lib/ld-linux.so.2 (0xb77ce000)
        libattr.so.1 => /lib/i386-linux-gnu/libattr.so.1 (0xb7497000)

Great. Time to increase our SSL security. A few notes:

To increase SSL security even further, we need to upgrade SSL and make it deny any compression.

To mitigate CRIME we need to disable ssl-compression
To mitigate BEAST and Lucky13 we need to use TLS 1.1 and above, no SSLv3 and no RC4
To mitigate BREACH we need to disable http-compression (gzip)

Also, to support strict transport security you need to add this to nginx.conf:

add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";

And httpd.conf:

LoadModule headers_module modules/mod_headers.so

<VirtualHost 192.168.99.99:80>
    Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"
</VirtualHost>


All right, let's get to configuring our OpenSSL:
root@zen-lb:~/hoard/Hoard/src# cd ~
root@zen-lb:~# mkdir openssl
root@zen-lb:~# cd openssl
root@zen-lb:~/openssl# apt-get source openssl
root@zen-lb:~/openssl# cd openssl-*
root@zen-lb:~/openssl/openssl-1.0.1e# quilt pop -a

Now, we need to disable compression, insecure ciphers, sslv2 (and sslv3):
root@zen-lb:~/openssl/openssl-1.0.1e# vi debian/rules
   CONFARGS  = -no-comp --prefix=/usr --openssldir=/usr/lib/ssl --libdir=lib/$(DEB_HOST_MULTIARCH) no-idea no-mdc2 no-rc5 no-zlib  enable-tlsext no-ssl2   #Include no-ssl3 for even better security.

Commit the changes:
root@zen-lb:~/openssl/openssl-1.0.1e# quilt push -a
root@zen-lb:~/openssl/openssl-1.0.1e# dpkg-source --commit
root@zen-lb:~/openssl/openssl-1.0.1e# debuild -uc -us
root@zen-lb:~/openssl/openssl-1.0.1e# cd ..
root@zen-lb:~/openssl# dpkg -i *ssl*.deb

Let's mark our openssl-related packages as non-upgradable. We do this so they don't get upgraded automatically to the vanilla one. This obviously means that we'll have to upgrade OpenSSL manually every time, just as we did earlier.
root@zen-lb:~/openssl# apt-mark hold libssl-dev libssl-doc libssl openssl libssl1.0.0 libssl1.0.0-dbg
root@zen-lb:~/openssl# reboot

Now, we need to upgrade Pound. Zen uses Pound for HTTP and HTTPS farms. Unfortunately, its version is old and does not support more modern SSL methods that are essential for SSL security. You need at least version 2.6f, which is very stable.

If you want the latest, check out if there are any bugs that may disrupt your service first. If you need elliptic curve cryptography support, I recommend the customized code by Joe Gooch, which can be found here. You need to make sure you select the "stage for upstream" branch. Joe Gooch's versions have some extra features as well, such as "DisableSSLv3","DisableTLSv10", "DisableTLSv11" and "DisableTLSv12".

In this example, I'm going to go ahead and use Pound 2.7c.
root@zen-lb:~# mkdir pound
root@zen-lb:~# cd pound
root@zen-lb:~/pound# wget https://fossies.org/linux/www/Pound-2.7c.tar.bz2
root@zen-lb:~/pound# bunzip2 Pound-2.7c.tar.bz2
root@zen-lb:~/pound# tar -xf Pound-2.7c.tar
root@zen-lb:~/pound# cd Pound-2.7c
root@zen-lb:~/pound/Pound-2.7c# ./configure
root@zen-lb:~/pound/Pound-2.7c# make
root@zen-lb:~/pound/Pound-2.7c# cp pound /usr/local/zenloadbalancer/app/pound/sbin/pound2.7c
root@zen-lb:~/pound/Pound-2.7c# cp poundctl /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.7c
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/pound /usr/local/zenloadbalancer/app/pound/sbin/pound2.5
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/poundctl /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.5
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/pound2.7c /usr/local/zenloadbalancer/app/pound/sbin/pound
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.7c /usr/local/zenloadbalancer/app/pound/sbin/poundctl
root@zen-lb:~/pound/Pound-2.7c# cd ~

Let's see if our pound is linked to the extra libraries we installed:
root@zen-lb:~# ldd /usr/local/zenloadbalancer/app/pound/sbin/pound
        linux-gate.so.1 =>  (0xb7751000)
        /usr/lib/libhoard.so (0xb770d000)
        libpcreposix.so.3 => /usr/lib/i386-linux-gnu/libpcreposix.so.3 (0xb7703000)
        libssl.so.1.0.0 => /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0 (0xb76aa000)
        libcrypto.so.1.0.0 => /usr/lib/i386-linux-gnu/i686/cmov/libcrypto.so.1.0.0 (0xb74ed000)
        libresolv.so.2 => /lib/i386-linux-gnu/i686/cmov/libresolv.so.2 (0xb74d9000)
        libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb74d5000)
        libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xb74af000)
        libtcmalloc.so.4 => /usr/lib/libtcmalloc.so.4 (0xb7447000)
        libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb742e000)
        libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb72ca000)
        libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb71de000)
        libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb71c1000)
        libpcre.so.3 => /lib/i386-linux-gnu/libpcre.so.3 (0xb7182000)
        /lib/ld-linux.so.2 (0xb7752000)

Great. Now, let's create a farm and test it out:


Note that if you're trying to find good ciphers to work with, Mozilla have some great recommendations here. There's also a great list here.

Well, everything looks good, so what about our new options? Let's manually edit our farmname_pound.cfg file:
root@zen-lb:~# vi /usr/local/zenloadbalancer/config/HTTPSTEST_pound.cfg
...
Threads  512
...
Cert "/usr/local/zenloadbalancer/config/zencert.pem"
        SSLHonorCipherOrder     1
        SSLAllowClientRenegotiation     0


The "Threads" variable exists in pound 2.7 and above so be careful about using it if you've downloaded a version in the 2.6 tree.

The SSLHonorCipherOrder 1 variable indicates that the order of the encryption processes that is defined in your Ciphers list must be respected by the web server, therefore the first cipher match found must be used.

The SSLAllowClientRenegotiation 0 variable indicates that no client renegotiation will be honored. When set to 0, no client renegotiation will be honored.  When 1, secure renegotiation will be honored.  When 2, insecure renegotiation will be honored.

Now, go and restart your farm from the UI. Test that everything works:

Great. Just in case, let's test that our SSL farm does not support insecure renegotiation (assuming my load balancer's external IP is 192.168.0.30):

root@zen-lb:~# openssl s_client -connect 192.168.0.30:443

and press "R" and return to renegotiate and then do "GET / HTTP /1.0". If you get a response, you've done something wrong.


 And that concludes part 2.

4 comments:

  1. Great post, I am interested in the libhoard.so library, did you do any stress test for knowing the difference of using it or not? Zen requires a lot of CPU in case of SSL but the memory is not the issue at the moment.

    Any feedback will be appreciated

    ReplyDelete
    Replies
    1. I did a few tests. I guess I should have taken a few screenshots to use for this series of posts. I'll make a post about this in the future I guess.

      Delete
  2. Replies
    1. Hi.

      Indeed this post is more than a year old which means quite a few things have changed since then. Notably you should now definitely add the no-ssl3 flag when building OpenSSL, and the Hoard and pound versions. In my opinion regarding the pound versions, Joe Gouch builds are still the ones to go for (https://github.com/goochjj/pound/ and look around for the ones flagged as "stage for upstream").

      An A+ score from Qualys is still attainable when using Zen Load Balancer if you follow these guidelines, your suported cipher suite is updated every so often and your cert is SHA2.

      Delete