All right. Time to get serious with this.
We 'll need these tools later on:
If PCRE, tcmalloc (from the Google perftools package) and/or Hoard are available Pound will link against them. This will provide a significant performance boost and is highly recommended. Guess what:
Add this to our /etc/profile so that the hoard library is loaded:
Test that it's loaded:
Great. Time to increase our SSL security. A few notes:
To increase SSL security even further, we need to upgrade SSL and make it deny any compression.
To mitigate CRIME we need to disable ssl-compression
To mitigate BEAST and Lucky13 we need to use TLS 1.1 and above, no SSLv3 and no RC4
To mitigate BREACH we need to disable http-compression (gzip)
Also, to support strict transport security you need to add this to nginx.conf:
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
And httpd.conf:
LoadModule headers_module modules/mod_headers.so
<VirtualHost 192.168.99.99:80>
Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"
</VirtualHost>
All right, let's get to configuring our OpenSSL:
Now, we need to disable compression, insecure ciphers, sslv2 (and sslv3):
Commit the changes:
Let's mark our openssl-related packages as non-upgradable. We do this so they don't get upgraded automatically to the vanilla one. This obviously means that we'll have to upgrade OpenSSL manually every time, just as we did earlier.
Now, we need to upgrade Pound. Zen uses Pound for HTTP and HTTPS farms. Unfortunately, its version is old and does not support more modern SSL methods that are essential for SSL security. You need at least version 2.6f, which is very stable.
If you want the latest, check out if there are any bugs that may disrupt your service first. If you need elliptic curve cryptography support, I recommend the customized code by Joe Gooch, which can be found here. You need to make sure you select the "stage for upstream" branch. Joe Gooch's versions have some extra features as well, such as "DisableSSLv3","DisableTLSv10", "DisableTLSv11" and "DisableTLSv12".
In this example, I'm going to go ahead and use Pound 2.7c.
Let's see if our pound is linked to the extra libraries we installed:
Great. Now, let's create a farm and test it out:
Note that if you're trying to find good ciphers to work with, Mozilla have some great recommendations here. There's also a great list here.
Well, everything looks good, so what about our new options? Let's manually edit our farmname_pound.cfg file:
The "Threads" variable exists in pound 2.7 and above so be careful about using it if you've downloaded a version in the 2.6 tree.
The SSLHonorCipherOrder 1 variable indicates that the order of the encryption processes that is defined in your Ciphers list must be respected by the web server, therefore the first cipher match found must be used.
The SSLAllowClientRenegotiation 0 variable indicates that no client renegotiation will be honored. When set to 0, no client renegotiation will be honored. When 1, secure renegotiation will be honored. When 2, insecure renegotiation will be honored.
Now, go and restart your farm from the UI. Test that everything works:
Great. Just in case, let's test that our SSL farm does not support insecure renegotiation (assuming my load balancer's external IP is 192.168.0.30):
and press "R" and return to renegotiate and then do "GET / HTTP /1.0". If you get a response, you've done something wrong.
And that concludes part 2.
root@zen-lb:~# apt-get update
root@zen-lb:~# apt-get upgradeWe 'll need these tools later on:
root@zen-lb:~# apt-get install build-essential devscripts m4 quilt debhelper zlib1g-dev bc gcc++ cmakeIf PCRE, tcmalloc (from the Google perftools package) and/or Hoard are available Pound will link against them. This will provide a significant performance boost and is highly recommended. Guess what:
root@zen-lb:~# apt-get install libpcrecpp0 libpcre3-dev libpcre3 libpcre++0 libpcre++-dev libtcmalloc-minimal4 libgoogle-perftools4 libgoogle-perftools-dev
root@zen-lb:~# mkdir hoard
root@zen-lb:~# cd hoard/
root@zen-lb:~/hoard# wget https://github.com/emeryberger/Hoard/releases/download/3.10/Hoard-3.10-source.tar.gz
root@zen-lb:~/hoard# gunzip Hoard-3.10-source.tar.gz
root@zen-lb:~/hoard# tar -xf Hoard-3.10-source.tar
root@zen-lb:~/hoard# cd Hoard/src
root@zen-lb:~/hoard/Hoard/src# make linux-gcc-x86
root@zen-lb:~/hoard/Hoard/src# cp libhoard.so /usr/lib/.Add this to our /etc/profile so that the hoard library is loaded:
root@zen-lb:~/hoard/Hoard/src# vi /etc/profileexport LD_PRELOAD=/usr/lib/libhoard.so
Test that it's loaded:
root@zen-lb:~# ldd /bin/ls linux-gate.so.1 => (0xb77cd000)
/usr/lib/libhoard.so (0xb7789000)
libselinux.so.1 => /lib/i386-linux-gnu/libselinux.so.1 (0xb7762000)
librt.so.1 => /lib/i386-linux-gnu/i686/cmov/librt.so.1 (0xb7758000)
libacl.so.1 => /lib/i386-linux-gnu/libacl.so.1 (0xb774e000)
libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb75ea000)
libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb75e6000)
libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb75cd000)
libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb74e0000)
libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xb74ba000)
libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb749d000)
/lib/ld-linux.so.2 (0xb77ce000)
libattr.so.1 => /lib/i386-linux-gnu/libattr.so.1 (0xb7497000)
Great. Time to increase our SSL security. A few notes:
To increase SSL security even further, we need to upgrade SSL and make it deny any compression.
To mitigate CRIME we need to disable ssl-compression
To mitigate BEAST and Lucky13 we need to use TLS 1.1 and above, no SSLv3 and no RC4
To mitigate BREACH we need to disable http-compression (gzip)
Also, to support strict transport security you need to add this to nginx.conf:
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
And httpd.conf:
LoadModule headers_module modules/mod_headers.so
<VirtualHost 192.168.99.99:80>
Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"
</VirtualHost>
All right, let's get to configuring our OpenSSL:
root@zen-lb:~/hoard/Hoard/src# cd ~
root@zen-lb:~# mkdir openssl
root@zen-lb:~# cd openssl
root@zen-lb:~/openssl# apt-get source openssl
root@zen-lb:~/openssl# cd openssl-*
root@zen-lb:~/openssl/openssl-1.0.1e# quilt pop -aNow, we need to disable compression, insecure ciphers, sslv2 (and sslv3):
root@zen-lb:~/openssl/openssl-1.0.1e# vi debian/rules CONFARGS = -no-comp --prefix=/usr --openssldir=/usr/lib/ssl --libdir=lib/$(DEB_HOST_MULTIARCH) no-idea no-mdc2 no-rc5 no-zlib enable-tlsext no-ssl2 #Include no-ssl3 for even better security.
Commit the changes:
root@zen-lb:~/openssl/openssl-1.0.1e# quilt push -a
root@zen-lb:~/openssl/openssl-1.0.1e# dpkg-source --commit
root@zen-lb:~/openssl/openssl-1.0.1e# debuild -uc -us
root@zen-lb:~/openssl/openssl-1.0.1e# cd ..
root@zen-lb:~/openssl# dpkg -i *ssl*.debLet's mark our openssl-related packages as non-upgradable. We do this so they don't get upgraded automatically to the vanilla one. This obviously means that we'll have to upgrade OpenSSL manually every time, just as we did earlier.
root@zen-lb:~/openssl# apt-mark hold libssl-dev libssl-doc libssl openssl libssl1.0.0 libssl1.0.0-dbg
root@zen-lb:~/openssl# rebootNow, we need to upgrade Pound. Zen uses Pound for HTTP and HTTPS farms. Unfortunately, its version is old and does not support more modern SSL methods that are essential for SSL security. You need at least version 2.6f, which is very stable.
If you want the latest, check out if there are any bugs that may disrupt your service first. If you need elliptic curve cryptography support, I recommend the customized code by Joe Gooch, which can be found here. You need to make sure you select the "stage for upstream" branch. Joe Gooch's versions have some extra features as well, such as "DisableSSLv3","DisableTLSv10", "DisableTLSv11" and "DisableTLSv12".
In this example, I'm going to go ahead and use Pound 2.7c.
root@zen-lb:~# mkdir pound
root@zen-lb:~# cd pound
root@zen-lb:~/pound# wget https://fossies.org/linux/www/Pound-2.7c.tar.bz2
root@zen-lb:~/pound# bunzip2 Pound-2.7c.tar.bz2
root@zen-lb:~/pound# tar -xf Pound-2.7c.tar
root@zen-lb:~/pound# cd Pound-2.7c
root@zen-lb:~/pound/Pound-2.7c# ./configure
root@zen-lb:~/pound/Pound-2.7c# make
root@zen-lb:~/pound/Pound-2.7c# cp pound /usr/local/zenloadbalancer/app/pound/sbin/pound2.7c
root@zen-lb:~/pound/Pound-2.7c# cp poundctl /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.7c
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/pound /usr/local/zenloadbalancer/app/pound/sbin/pound2.5
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/poundctl /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.5
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/pound2.7c /usr/local/zenloadbalancer/app/pound/sbin/pound
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.7c /usr/local/zenloadbalancer/app/pound/sbin/poundctl
root@zen-lb:~/pound/Pound-2.7c# cd ~Let's see if our pound is linked to the extra libraries we installed:
root@zen-lb:~# ldd /usr/local/zenloadbalancer/app/pound/sbin/pound linux-gate.so.1 => (0xb7751000)
/usr/lib/libhoard.so (0xb770d000)
libpcreposix.so.3 => /usr/lib/i386-linux-gnu/libpcreposix.so.3 (0xb7703000)
libssl.so.1.0.0 => /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0 (0xb76aa000)
libcrypto.so.1.0.0 => /usr/lib/i386-linux-gnu/i686/cmov/libcrypto.so.1.0.0 (0xb74ed000)
libresolv.so.2 => /lib/i386-linux-gnu/i686/cmov/libresolv.so.2 (0xb74d9000)
libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb74d5000)
libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xb74af000)
libtcmalloc.so.4 => /usr/lib/libtcmalloc.so.4 (0xb7447000)
libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb742e000)
libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb72ca000)
libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb71de000)
libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb71c1000)
libpcre.so.3 => /lib/i386-linux-gnu/libpcre.so.3 (0xb7182000)
/lib/ld-linux.so.2 (0xb7752000)
Great. Now, let's create a farm and test it out:
Note that if you're trying to find good ciphers to work with, Mozilla have some great recommendations here. There's also a great list here.
Well, everything looks good, so what about our new options? Let's manually edit our farmname_pound.cfg file:
root@zen-lb:~# vi /usr/local/zenloadbalancer/config/HTTPSTEST_pound.cfg...
Threads 512
...
Cert "/usr/local/zenloadbalancer/config/zencert.pem"
SSLHonorCipherOrder 1
SSLAllowClientRenegotiation 0
The "Threads" variable exists in pound 2.7 and above so be careful about using it if you've downloaded a version in the 2.6 tree.
The SSLHonorCipherOrder 1 variable indicates that the order of the encryption processes that is defined in your Ciphers list must be respected by the web server, therefore the first cipher match found must be used.
The SSLAllowClientRenegotiation 0 variable indicates that no client renegotiation will be honored. When set to 0, no client renegotiation will be honored. When 1, secure renegotiation will be honored. When 2, insecure renegotiation will be honored.
Now, go and restart your farm from the UI. Test that everything works:
root@zen-lb:~# openssl s_client -connect 192.168.0.30:443and press "R" and return to renegotiate and then do "GET / HTTP /1.0". If you get a response, you've done something wrong.
And that concludes part 2.
Great post, I am interested in the libhoard.so library, did you do any stress test for knowing the difference of using it or not? Zen requires a lot of CPU in case of SSL but the memory is not the issue at the moment.
ReplyDeleteAny feedback will be appreciated
I did a few tests. I guess I should have taken a few screenshots to use for this series of posts. I'll make a post about this in the future I guess.
DeleteHi,
ReplyDeleteAny update this post?
Hi.
DeleteIndeed this post is more than a year old which means quite a few things have changed since then. Notably you should now definitely add the no-ssl3 flag when building OpenSSL, and the Hoard and pound versions. In my opinion regarding the pound versions, Joe Gouch builds are still the ones to go for (https://github.com/goochjj/pound/ and look around for the ones flagged as "stage for upstream").
An A+ score from Qualys is still attainable when using Zen Load Balancer if you follow these guidelines, your suported cipher suite is updated every so often and your cert is SHA2.