Wednesday, July 9, 2014

Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 2

All right. Time to get serious with this.
root@zen-lb:~# apt-get update
root@zen-lb:~# apt-get upgrade

We 'll need these tools later on:
root@zen-lb:~# apt-get install build-essential devscripts m4 quilt debhelper zlib1g-dev bc gcc++ cmake

If PCRE, tcmalloc (from the Google perftools package) and/or Hoard are available Pound will link against them. This will provide a significant performance boost and is highly recommended. Guess what:
root@zen-lb:~# apt-get install libpcrecpp0 libpcre3-dev libpcre3 libpcre++0 libpcre++-dev libtcmalloc-minimal4 libgoogle-perftools4 libgoogle-perftools-dev
root@zen-lb:~# mkdir hoard
root@zen-lb:~# cd hoard/
root@zen-lb:~/hoard# wget https://github.com/emeryberger/Hoard/releases/download/3.10/Hoard-3.10-source.tar.gz
root@zen-lb:~/hoard# gunzip Hoard-3.10-source.tar.gz 
root@zen-lb:~/hoard# tar -xf Hoard-3.10-source.tar 
root@zen-lb:~/hoard# cd Hoard/src
root@zen-lb:~/hoard/Hoard/src# make linux-gcc-x86
root@zen-lb:~/hoard/Hoard/src# cp libhoard.so /usr/lib/.

Add this to our /etc/profile so that the hoard library is loaded:
root@zen-lb:~/hoard/Hoard/src# vi /etc/profile
export LD_PRELOAD=/usr/lib/libhoard.so

Test that it's loaded:
root@zen-lb:~# ldd /bin/ls
        linux-gate.so.1 =>  (0xb77cd000)
        /usr/lib/libhoard.so (0xb7789000)
        libselinux.so.1 => /lib/i386-linux-gnu/libselinux.so.1 (0xb7762000)
        librt.so.1 => /lib/i386-linux-gnu/i686/cmov/librt.so.1 (0xb7758000)
        libacl.so.1 => /lib/i386-linux-gnu/libacl.so.1 (0xb774e000)
        libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb75ea000)
        libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb75e6000)
        libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb75cd000)
        libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb74e0000)
        libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xb74ba000)
        libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb749d000)
        /lib/ld-linux.so.2 (0xb77ce000)
        libattr.so.1 => /lib/i386-linux-gnu/libattr.so.1 (0xb7497000)

Great. Time to increase our SSL security. A few notes:

To increase SSL security even further, we need to upgrade SSL and make it deny any compression.

To mitigate CRIME we need to disable ssl-compression
To mitigate BEAST and Lucky13 we need to use TLS 1.1 and above, no SSLv3 and no RC4
To mitigate BREACH we need to disable http-compression (gzip)

Also, to support strict transport security you need to add this to nginx.conf:

add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";

And httpd.conf:

LoadModule headers_module modules/mod_headers.so

<VirtualHost 192.168.99.99:80>
    Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"
</VirtualHost>


All right, let's get to configuring our OpenSSL:
root@zen-lb:~/hoard/Hoard/src# cd ~
root@zen-lb:~# mkdir openssl
root@zen-lb:~# cd openssl
root@zen-lb:~/openssl# apt-get source openssl
root@zen-lb:~/openssl# cd openssl-*
root@zen-lb:~/openssl/openssl-1.0.1e# quilt pop -a

Now, we need to disable compression, insecure ciphers, sslv2 (and sslv3):
root@zen-lb:~/openssl/openssl-1.0.1e# vi debian/rules
   CONFARGS  = -no-comp --prefix=/usr --openssldir=/usr/lib/ssl --libdir=lib/$(DEB_HOST_MULTIARCH) no-idea no-mdc2 no-rc5 no-zlib  enable-tlsext no-ssl2   #Include no-ssl3 for even better security.

Commit the changes:
root@zen-lb:~/openssl/openssl-1.0.1e# quilt push -a
root@zen-lb:~/openssl/openssl-1.0.1e# dpkg-source --commit
root@zen-lb:~/openssl/openssl-1.0.1e# debuild -uc -us
root@zen-lb:~/openssl/openssl-1.0.1e# cd ..
root@zen-lb:~/openssl# dpkg -i *ssl*.deb

Let's mark our openssl-related packages as non-upgradable. We do this so they don't get upgraded automatically to the vanilla one. This obviously means that we'll have to upgrade OpenSSL manually every time, just as we did earlier.
root@zen-lb:~/openssl# apt-mark hold libssl-dev libssl-doc libssl openssl libssl1.0.0 libssl1.0.0-dbg
root@zen-lb:~/openssl# reboot

Now, we need to upgrade Pound. Zen uses Pound for HTTP and HTTPS farms. Unfortunately, its version is old and does not support more modern SSL methods that are essential for SSL security. You need at least version 2.6f, which is very stable.

If you want the latest, check out if there are any bugs that may disrupt your service first. If you need elliptic curve cryptography support, I recommend the customized code by Joe Gooch, which can be found here. You need to make sure you select the "stage for upstream" branch. Joe Gooch's versions have some extra features as well, such as "DisableSSLv3","DisableTLSv10", "DisableTLSv11" and "DisableTLSv12".

In this example, I'm going to go ahead and use Pound 2.7c.
root@zen-lb:~# mkdir pound
root@zen-lb:~# cd pound
root@zen-lb:~/pound# wget https://fossies.org/linux/www/Pound-2.7c.tar.bz2
root@zen-lb:~/pound# bunzip2 Pound-2.7c.tar.bz2
root@zen-lb:~/pound# tar -xf Pound-2.7c.tar
root@zen-lb:~/pound# cd Pound-2.7c
root@zen-lb:~/pound/Pound-2.7c# ./configure
root@zen-lb:~/pound/Pound-2.7c# make
root@zen-lb:~/pound/Pound-2.7c# cp pound /usr/local/zenloadbalancer/app/pound/sbin/pound2.7c
root@zen-lb:~/pound/Pound-2.7c# cp poundctl /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.7c
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/pound /usr/local/zenloadbalancer/app/pound/sbin/pound2.5
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/poundctl /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.5
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/pound2.7c /usr/local/zenloadbalancer/app/pound/sbin/pound
root@zen-lb:~/pound/Pound-2.7c# cp /usr/local/zenloadbalancer/app/pound/sbin/poundctl2.7c /usr/local/zenloadbalancer/app/pound/sbin/poundctl
root@zen-lb:~/pound/Pound-2.7c# cd ~

Let's see if our pound is linked to the extra libraries we installed:
root@zen-lb:~# ldd /usr/local/zenloadbalancer/app/pound/sbin/pound
        linux-gate.so.1 =>  (0xb7751000)
        /usr/lib/libhoard.so (0xb770d000)
        libpcreposix.so.3 => /usr/lib/i386-linux-gnu/libpcreposix.so.3 (0xb7703000)
        libssl.so.1.0.0 => /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0 (0xb76aa000)
        libcrypto.so.1.0.0 => /usr/lib/i386-linux-gnu/i686/cmov/libcrypto.so.1.0.0 (0xb74ed000)
        libresolv.so.2 => /lib/i386-linux-gnu/i686/cmov/libresolv.so.2 (0xb74d9000)
        libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb74d5000)
        libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xb74af000)
        libtcmalloc.so.4 => /usr/lib/libtcmalloc.so.4 (0xb7447000)
        libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb742e000)
        libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb72ca000)
        libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb71de000)
        libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb71c1000)
        libpcre.so.3 => /lib/i386-linux-gnu/libpcre.so.3 (0xb7182000)
        /lib/ld-linux.so.2 (0xb7752000)

Great. Now, let's create a farm and test it out:


Note that if you're trying to find good ciphers to work with, Mozilla have some great recommendations here. There's also a great list here.

Well, everything looks good, so what about our new options? Let's manually edit our farmname_pound.cfg file:
root@zen-lb:~# vi /usr/local/zenloadbalancer/config/HTTPSTEST_pound.cfg
...
Threads  512
...
Cert "/usr/local/zenloadbalancer/config/zencert.pem"
        SSLHonorCipherOrder     1
        SSLAllowClientRenegotiation     0


The "Threads" variable exists in pound 2.7 and above so be careful about using it if you've downloaded a version in the 2.6 tree.

The SSLHonorCipherOrder 1 variable indicates that the order of the encryption processes that is defined in your Ciphers list must be respected by the web server, therefore the first cipher match found must be used.

The SSLAllowClientRenegotiation 0 variable indicates that no client renegotiation will be honored. When set to 0, no client renegotiation will be honored.  When 1, secure renegotiation will be honored.  When 2, insecure renegotiation will be honored.

Now, go and restart your farm from the UI. Test that everything works:

Great. Just in case, let's test that our SSL farm does not support insecure renegotiation (assuming my load balancer's external IP is 192.168.0.30):

root@zen-lb:~# openssl s_client -connect 192.168.0.30:443

and press "R" and return to renegotiate and then do "GET / HTTP /1.0". If you get a response, you've done something wrong.


 And that concludes part 2.

Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 1

I'm a bit partial to Zen Load Balancer. As a matter of a fact, I love it. It has many, many things ready to go from the start.

And it being just a Debian distro with the zenloadbalancer package on top, there's a lot you can do to customize it. The first thing we need to do is get rid of this:



My system has 16GB of memory but memory reported is just 3GB? Yup.
See Zen Load Balancer is a 32-bit app and it is distributed with a 32-bit Debian distro.

Assuming we have a 64-bit system with more memory installed we'll need to upgrade the kernel to a PAE one. This is both a performance and a security enhancement. It will allow us to use more memory and will also enable NX protection (provided that our BIOS and CPU support it too), as the NX bit works on the 63rd bit of the address.

Editing our repos first:
root@zen-lb:~# vi /etc/apt/sources.list
#official repository for Debian
deb http://ftp.debian.org/debian/ stable main non-free
deb-src http://ftp.debian.org/debian/ stable main non-free
deb http://security.debian.org/ stable/updates main
deb-src http://security.debian.org/ stable/updates main
#official repository for Zen Load Balancer Updates
deb http://zenloadbalancer.sourceforge.net/apt/x86 v3/

#Let's add this repo as well to do a moderate PAE upgrade at first
deb http://security.debian.org/debian-security squeeze/updates main

Let's try to upgrade our kernel now:
root@zen-lb:~# apt-get update
....
....
....
Reading package lists... Done
W: GPG error: http://ftp.debian.org stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B48AD6246925553 NO_PUBKEY 6FB2A1C265FFB764
W: GPG error: http://security.debian.org stable/updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B48AD6246925553

Yeah ok, classic Debian thing:
root@zen-lb:~# gpg --keyserver pgpkeys.mit.edu --recv-key 6FB2A1C265FFB764
root@zen-lb:~# gpg -a --export 6FB2A1C265FFB764 | apt-key add -
root@zen-lb:~# gpg --keyserver pgpkeys.mit.edu --recv-key 8B48AD6246925553
root@zen-lb:~# gpg -a --export 8B48AD6246925553 | apt-key add -
root@zen-lb:~# apt-get update
Fetched 548 kB in 1s (444 kB/s)
Reading package lists... Done

One more time for the world:
root@zen-lb:~# apt-cache search linux-image
linux-image-2.6-486 - Linux for older PCs (dummy package)
linux-image-2.6-686 - Linux for modern PCs (dummy package)
linux-image-2.6-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
linux-image-2.6-686-pae - Linux for modern PCs (dummy package)
linux-image-2.6-amd64 - Linux for 64-bit PCs (dummy package)
linux-image-486 - Linux for older PCs (meta-package)
linux-image-686 - Linux for modern PCs (dummy package)
linux-image-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
linux-image-686-pae - Linux for modern PCs (meta-package)
linux-image-amd64 - Linux for 64-bit PCs (meta-package)
linux-image-rt-686-pae - Linux for modern PCs (meta-package), PREEMPT_RT
linux-image-3.2.0-4-486 - Linux 3.2 for older PCs
linux-image-3.2.0-4-686-pae - Linux 3.2 for modern PCs
linux-image-3.2.0-4-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-686-pae
linux-image-3.2.0-4-amd64 - Linux 3.2 for 64-bit PCs
linux-image-3.2.0-4-rt-686-pae - Linux 3.2 for modern PCs, PREEMPT_RT
linux-image-3.2.0-4-rt-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-rt-686-pae
linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486
linux-headers-2.6.32-5-686 - Header files for Linux 2.6.32-5-686
linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem
linux-headers-2.6.32-5-amd64 - Header files for Linux 2.6.32-5-amd64
linux-headers-2.6.32-5-openvz-686 - Header files for Linux 2.6.32-5-openvz-686
linux-headers-2.6.32-5-vserver-686 - Header files for Linux 2.6.32-5-vserver-686
linux-headers-2.6.32-5-vserver-686-bigmem - Header files for Linux 2.6.32-5-vserver-686-bigmem
linux-headers-2.6.32-5-xen-686 - Header files for Linux 2.6.32-5-xen-686
linux-image-2.6.32-5-486 - Linux 2.6.32 for old PCs
linux-image-2.6.32-5-686 - Linux 2.6.32 for modern PCs
linux-image-2.6.32-5-686-bigmem - Linux 2.6.32 for PCs with 4GB+ RAM
linux-image-2.6.32-5-686-bigmem-dbg - Debugging infos for Linux 2.6.32-5-686-bigmem
linux-image-2.6.32-5-amd64 - Linux 2.6.32 for 64-bit PCs
linux-image-2.6.32-5-openvz-686 - Linux 2.6.32 for modern PCs, OpenVZ support
linux-image-2.6.32-5-openvz-686-dbg - Debugging infos for Linux 2.6.32-5-openvz-686
linux-image-2.6.32-5-vserver-686 - Linux 2.6.32 for modern PCs, Linux-VServer support
linux-image-2.6.32-5-vserver-686-bigmem - Linux 2.6.32 for PCs with 4GB+ RAM, Linux-VServer support
linux-image-2.6.32-5-vserver-686-bigmem-dbg - Debugging infos for Linux 2.6.32-5-vserver-686-bigmem
linux-image-2.6.32-5-xen-686 - Linux 2.6.32 for modern PCs, Xen dom0 support
linux-image-2.6.32-5-xen-686-dbg - Debugging infos for Linux 2.6.32-5-xen-686

Right, let's be conservative and upgrade to a 2.6 PAE kernel, we'll do a major upgrade later:
root@zen-lb:~# uname -a
Linux zen-lb 2.6.32-5-686 #1 SMP Wed Jan 12 04:01:41 UTC 2011 i686 GNU/Linux
root@zen-lb:~# apt-get install linux-image-2.6.32-5-686-bigmem
Get:1 http://ftp.debian.org/debian/ stable/main linux-base all 3.5 [34.3 kB]
Get:2 http://security.debian.org/debian-security/ squeeze/updates/main linux-image-2.6.32-5-686-bigmem i386 2.6.32-48squeeze6 [27.6 MB]
Get:3 http://ftp.debian.org/debian/ stable/main firmware-linux-free all 3.2 [20.7 kB]
Fetched 27.7 MB in 20s (1,366 kB/s)
Preconfiguring packages ...
(Reading database ... 18065 files and directories currently installed.)
Preparing to replace linux-base 2.6.32-30 (using .../linux-base_3.5_all.deb) ...
Unpacking replacement linux-base ...
Selecting previously deselected package linux-image-2.6.32-5-686-bigmem.
Unpacking linux-image-2.6.32-5-686-bigmem (from .../linux-image-2.6.32-5-686-bigmem_2.6.32-48squeeze6_i386.deb) ...
Selecting previously deselected package firmware-linux-free.
Unpacking firmware-linux-free (from .../firmware-linux-free_3.2_all.deb) ...
Processing triggers for man-db ...
Setting up linux-base (3.5) ...
Setting up linux-image-2.6.32-5-686-bigmem (2.6.32-48squeeze6) ...
Running depmod.
Running update-initramfs.
update-initramfs: Generating /boot/initrd.img-2.6.32-5-686-bigmem
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 2.6.32-5-686-bigmem /boot/vmlinuz-2.6.32-5-686-bigmem
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 2.6.32-5-686-bigmem /boot/vmlinuz-2.6.32-5-686-bigmem
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-5-686-bigmem
Found initrd image: /boot/initrd.img-2.6.32-5-686-bigmem
Found linux image: /boot/vmlinuz-2.6.32-5-686
Found initrd image: /boot/initrd.img-2.6.32-5-686
done
Setting up firmware-linux-free (3.2) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-2.6.32-5-686-bigmem
root@zen-lb:~# reboot

All right, so did it work?

Looks good. And what about NX?
root@zen-lb:~# dmesg | grep ".*NX.*protection"
[    0.000000] NX (Execute Disable) protection: active

Cool. Let's go on then. Let's do a distro upgrade from Squeeze to Wheezy:
root@zen-lb:~# apt-get dist-upgrade
root@zen-lb:~# reboot

Indeed:
root@zen-lb:~# cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"
NAME="Debian GNU/Linux"
VERSION_ID="7"
VERSION="7 (wheezy)"
ID=debian
ANSI_COLOR="1;31"
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support/"
BUG_REPORT_URL="http://bugs.debian.org/"
root@zen-lb:~# cat /proc/version
Linux version 3.2.0-4-686-pae (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.60-1+deb7u1

The first thing I'm going to do is tune my filesystem. To be honest, this is just a load balancer so I can afford to lose a few seconds of logs if the power goes down.
root@zen-lb:~# vi /etc/fstab
proc            /proc           proc    defaults        0       0
# / was on /dev/sdb3 during installation
UUID=b6016824-536e-43bc-8f1f-fbfd2fab146d /               ext4    noatime,nodiratime,nobarrier,nobh,commit=120,data=writeback,journal_async_commit,errors=remount-ro 0       1
# /boot was on /dev/sdb1 during installation
UUID=6d4bd9ca-ba29-4700-b90c-07c614d79f0e /boot           ext4    defaults        0       2
# swap was on /dev/sdb2 during installation
UUID=3cb4eb4d-b0ab-4b60-825b-fc0224356580 none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

Go to single user mode and tune my root filesystem (mine is on /dev/sdb3):
root@zen-lb:~# init 1
root@zen-lb:~# tune2fs -O dir_index /dev/sdb3
root@zen-lb:~# umount -a
root@zen-lb:~# e2fsck -D /dev/sdb3

And now increase the number of open files limit:
root@zen-lb:~# vi /etc/security/limits.conf
....
* soft nofile 65536
* hard nofile 65536
# End of file

When regular users log in, they get an open files warning, let's correct it by uncommenting these few lines of code in /etc/profile:
root@zen-lb:~# vi /etc/profile
....
#if [ -f /etc/sysctl.conf ]; then
# FILEMAX=`grep "^fs.file-max.*=" /etc/sysctl.conf | awk -F'=' '{printf $2}'`
# if [ "$FILEMAX" != "" ]; then
#  ulimit -n $FILEMAX
# fi
#fi

Finally, let's update our repos to correctly receive wheezy updates:
root@zen-lb:~# vi /etc/apt/sources.list
#official repository for Debian
deb http://ftp.debian.org/debian wheezy main contrib non-free
deb-src http://ftp.debian.org/debian wheezy main contrib non-free
deb http://ftp.debian.org/debian wheezy-updates main contrib non-free

deb http://http.debian.net/debian wheezy main contrib non-free
deb-src http://http.debian.net/debian wheezy main contrib non-free

deb http://http.debian.net/debian wheezy-updates main contrib non-free
deb-src http://http.debian.net/debian wheezy-updates main contrib non-free

deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free

#official repository for Zen Load Balancer Updates
deb http://zenloadbalancer.sourceforge.net/apt/x86 v3/

All right, I guess we didn't do that much, but it's enough to call the end of Part 1.

Monday, June 23, 2014

XenServer 6.2 Unable to find partition containing kernel

Sometimes XenServer can be a real pain in the neck.

Here's what I ran into a little earlier:

I upgrade my VM's kernel and reboot. Wait for a while, but the VM doesn't appear to come up. So I go to my Logs and lo and behold:



Ugh, wat?

OK, let's see if we can start it manually, worst case scenario is the command line providing us with more info, right?

[root@xen]# xe vm-start vm="My VM's Name"
The bootloader returned an error
vm: .... (My VM's Name)
msg: Unable to find partition containing kernel

Oh, how informative. Luckily XenServer provides us with an off-line VM boot editor. That should do the trick:

[root@xen]# xe-edit-bootloader -n "My VM's Name" -p 1
Plugging VBD: 
Creating dom0 VBD: ...
add map ...1 (252:5): 0 1048576 linear /dev/sm/backend/.../... 2048
add map ...2 (252:6): 0 33554432 linear /dev/sm/backend/.../... 1050624
add map ...3 (252:7): 0 1886386176 linear /dev/sm/backend/.../... 34605056
Waiting for /dev/mapper/...1: .....Device /dev/mapper/...1 not found.
You must specify the correct partition number with -p
Unplugging VBD: . done

This should have worked. It's usually as good as logging into the VM and editing the GRUB menu.lst yourself. What on earth happened here? Well, I'm pretty sure I know my systems and my partitions so I'm sure the /boot partition is the first one!

What now? Well, for sure we'll need to boot from a rescue disk. So, make sure your NFS ISO library is online or you have a storage already handy that has a rescue CD .iso. No? We don't? Well, no reason to panic. We'll just create a directory named "/rescue/ISOs", download a Debian Live CD there and create an SR with the name label "RESCUE" that points to that directory:

[root@xen]# mkdir -p /rescue/ISOs
[root@xen]# cd /rescue/ISOs
[root@xen]# wget http://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/debian-live-7.5.0-amd64-rescue.iso
[root@xen]# xe sr-create name-label=RESCUE type=iso device-config:legacy_mode=true device-config:location=/rescue/ISOs content-type=iso
303bd588-c675-2bc0-8982-8a691141226a 

Please note that you need to make sure you create this SR on a partition with plenty of disk space, the Dom0's default partition is just too small and therefore I do not recommend having any rescue ISOs on it.

Cool. Now let's select that ISO as our DVD disk on our VM and finally boot off of it. The latter task can be done by going to VM -> Start/Shut Down -> Start in Recovery Mode.

Now that everything's started up, let's see what's going on. As I already mentioned, my /boot partition is my first partition so let's mount it and inspect the grub.conf file:

root@debian:~# cat /proc/partitions
major minor  #blocks  name

 202        0  960495616 xvda
 202        1     524288 xvda1
 202        2   16777216 xvda2
 202        3  943193088 xvda3
  11        0     696320 sr0
   7        0     548992 loop0 
root@debian:~# mkdir /a
root@debian:~# mount /dev/xvda1 /a
root@debian:~# vi /a/grub/grub.conf 
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/xvda3
#          initrd /initrd-[generic-]version.img
#boot=/dev/xvda
default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Oracle Linux Server (3.8.13-35.1.2.el6uek.x86_64)
        root (hd0,0)
        kernel /vmlinuz-3.8.13-35.1.2.el6uek.x86_64 ro root=UUID=e9b3edd9-15e0-4cfd-a8fa-7dc24f6aeefa rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD console=hvc0  KEYTABLE=us SYSFONT=latarcyrheb-sun16  rd_NO_LVM rd_NO_DM rhgb quiet
        initrd /initramfs-3.8.13-35.1.2.el6uek.x86_64.img

Now, we have two options really:

a) Solving this through XenServer. We would have to issue:

xe vm-param-set uuid=vm-uuid PV-bootloader-args="--kernel=/vmlinuz-3.8.13-35.1.2.el6uek.x86_64 --ramdisk=/initramfs-3.8.13-35.1.2.el6uek.x86_64.img"
xe vm-param-set uuid=vm-uuid PV-args="root=root-device  ro quiet"

Since I would have to change this every time I upgrade the kernel and I really want to find what went wrong, I'll pass this option for now.

b) Trying to debug what's wrong with grub.conf.

Since the kernel is too recent to not support XenServer, it should just be a matter of patience to debug it.

Here's what's usually wrong with grub.conf/menu.lst:
i  ) The root(hdx,y) is wrong:
          x should point to the hard drive number where our boot partition is located at;
          y should point to the partition number of our boot partition.
   In this case, root(hd0,0) is correct.
ii ) The paths of vmlinuz-... and/or initramfs-... are wrong.
         The paths should be relative to the partition root directory. So for example if the boot directory is in a dedicated partition, it should be /vmlinuz-... and /initramfs-... but if the boot directory is in the same partition as the linux root (/) directory it should be /boot/vmlinuz-... and /boot/initramfs... if that explanation makes sense.
iii) The root directive is wrong.
          Here I mean the root directive that defines the linux root (/) directory, and not the boot partition which has been already declared with the root(hdx,y) statement. It could be root=/dev/xvda3 for instance. In my case it is root=UUID=e9b3edd9-15e0-4cfd-a8fa-7dc24f6aeefa.

The three cases above can be easily examined with a simple ls command on the /a directory and a blkid /dev/xvda3 or ls -l /dev/disk/by-uuid to find if the UUID of the device that hosts our root (/) directory is the correct one.

Another case is for the partition to have been corrupted, so we just umount /a and fsck /dev/xvda1    

In my case, as you can see the error was in the default setting. Changing this to 0 did the trick. Default signifies which item in the menu is the one that will boot after the user-interaction timeout occurs. The count starts from 0 and not from 1 so my system couldn't boot. What happened is that the OS provider had decided to change the default kernel order, making a mess out of my server.

Save changes, shutdown, reboot. Should be fine.

Friday, June 6, 2014

Subnetting made easy Part 2

Class A:
0.0.0.0     - 127.255.255.255 default subnet mask: 255.0.0.0 (or /8, 8 network bits and 24 host bits)
Class B:
128.0.0.0 - 191.255.255.255 default subnet mask: 255.255.0.0 (or /16, 16 network bits and 16 host bits)
Class C:
192.0.0.0 - 223.255.255.255 default subnet mask: 255.255.255.0 (or /24, 24 network bits and 8 host bits)
Class D: (Used for multicasting, not used for IP addressing)
224.0.0.0 - 239.255.255.255
Class E: (Unused)
240.0.0.0 - 255.255.255.255

Subnetting based on the number of hosts:

Example A:

Say that our assigned network is 213.213.213.0 and we need to divide it into networks that have 30 hosts each.

Let's consult a table for our decimal to binary conversion to find what 30 is in binary. We work with 8 bits and we start from 0, so we work from 20 to 27:

2726252423222120
1286432168421
00011110

That means we need 5 bits to represent number 30 (24 is the 5th bit, we start from 0) and therefore we need 5 bits for our hosts. 52=32, and the number of the actual usable hosts is 52-2=30.

213.213.213.0 is a Class C network.

Class C subnet mask: 255.255.255.0 (/24) or in binary:   11111111 11111111 11111111 00000000

Our subnet mask will be the default mask with the number of network bits that we will need to have in order to have the number of hosts we desire.

Class C subnet mask: 11111111 11111111 11111111 00000000
                                                                                  ^^^ we need 5 host bits, so we'll need to turn the last 5 bits to 0; the rest will be assigned as network bits by turning them to 1.                    

Our     subnet mask: 11111111 11111111 11111111 11100000
 In other words     :      255          255       255           224   (/27, since we have 27 network bits enabled)

So now, the number of our hosts will be the remaining host bits. In our case 32. The network bits are 3, so 23=8 networks.

The number of usable hosts in our network though will be the remaining host bits minus the network
address and the broadcast address, so 2 to the power of our remaining host bits minus 2.

So, in our case 25-2=32-2=30. We will have 30 hosts -exactly as many as we needed- in 8 networks.

A faster way to find this number is with what Cisco refers to as the interesting part. The interesting part is
the part of the subnet mask which is not 255. We subtract this number from 256 and that will be our IP address increment.

This in our case is the fourth part of the IP address. Also, In our case, 256-224=32.

So our first IP allocation will be 213.213.213.32. Just write the increments and fill the spaces in-between.

213.213.213.0
213.213.213.32
213.213.213.64
213.213.213.96
etc.
So our network is:

213.213.213.0 - 213.213.213.31
213.213.213.32 - 213.213.213.63
213.213.213.64 - 213.213.213.95
etc.
.....

Example B:

We are given the network 130.130.0.0 and we want to divide it into networks with 600 hosts each.

Let's consult a table for our decimal to binary conversion to find how 600 is represented:

The sum of our 8-bit table (128 to 1) amounts to 255 (obviously). 8 bits are not enough. We need more entries:

5122561286432168421
1001011000


So we need 10 bits for our hosts. 102-2=1024 number of hosts, 1022 usable hosts.

This is a Class B network so its network mask looks like this:

11111111 11111111 00000000 00000000


We need 10 host bits, so we'll need to turn the last 10 bits to 0; the rest will be assigned as network bits by turning them to 1. And now our netmask looks like this:

11111111 11111111 11111100 00000000 (/22)
   255          255           252          0

As we already said, we need 10 bits for our hosts. 102-2=1024 number of hosts, 1022 usable hosts.
The network bits are 6, so 26=64 networks.

256-252=4. So this is our network:

130.130.0.0
130.130.4.0
130.130.8.0
etc.

and our IP ranges are:

130.130.0.0 - 130.130.3.255
130.130.4.0 - 130.130.7.255
etc.
.....

Example C:

We have the network 10.0.0.0 and we want to split it into networks with 96 hosts each.

Let's consult a table for our decimal to binary conversion to find how 96 is represented:

12864.................
01.................

No need to calculate any more, we are sure that this is the correct bit, we go over after that
So we need 7 bits for our hosts. 27=128 hosts, 128-2=126 usable addresses.

This is a Class A network so its netmask looks like ths:

11111111 00000000 00000000 00000000

We need to have 7 host bits, so our netmask should look like this:

11111111 11111111 11111111 10000000 (/25)
   255           255         255         128

So 27=128 hosts, 128-2=126 usable addresses, 217=131072 networks.

256-128=128. Our IP ranges are:

10.0.0.0 - 10.0.0.127
10.0.0.128 - 10.0.0.255
10.0.1.0 - 10.0.1.127
etc.

Now that we've gone through the basics and learned how to subnet, we need to note:
It's always a good idea to add one when trying to subnet based on hosts just in case our number is on a binary calculation boundary (e.g. 128,64,32,16,8,4,2,1).

Let's go over an example again with our new method:

Say that our assigned network is 213.213.213.0 and we need to divide it into networks that have 32 hosts each.

We need to add one so instead of calculating for 32, we calculate for 32+1=33:

Let's consult a table for our decimal to binary conversion to find what 33 is in binary.

1286432168421
00100001

We need 6 bits to represent number 33. 62=36, and the number of the actual usable hosts is 62-2=34.

213.213.213.0 is a Class C network.

Class C subnet mask: 255.255.255.0 (/24) or in binary:   11111111 11111111 11111111 00000000

Our subnet mask will be the default mask with the number of network bits that we will need to have in order to have the number of hosts we desire.

Class C subnet mask: 11111111 11111111 11111111 00000000
                                                                                                     
Our subnet mask:       11111111 11111111 11111111 11000000
In other words:             255          255          255         192   (/26, since we have 26 network bits enabled)

The network bits are 2, so 22=4 networks.

The number of usable hosts in our network will be the remaining host bits minus the network address and the broadcast address, so 2 to the power of our remaining host bits minus 2. As calculated earlier, 62=36, and the number of the actual usable hosts is 62-2=34.


This in our case is the fourth part of the IP address. Also, In our case, 256-192=64.

So our first IP allocation will be 213.213.213.64. Just write the increments and fill the spaces in-between.

213.213.213.0
213.213.213.64
213.213.213.128
213.213.213.192

So our network is this:

213.213.213.0 - 213.213.213.63
213.213.213.64 - 213.213.213.127
213.213.213.128 - 213.213.213.191
213.213.213.192 - 213.213.213.255

If we had done the calculation without adding one, we would have provisioned for 30 hosts (5 bits,25=32,30 usable hosts)

Sunday, May 25, 2014

Subnetting made easy Part 1

Class A:
0.0.0.0     - 127.255.255.255 default subnet mask: 255.0.0.0 (or /8, 8 network bits and 24 host bits)
Class B:
128.0.0.0 - 191.255.255.255 default subnet mask: 255.255.0.0 (or /16, 16 network bits and 16 host bits)
Class C:
192.0.0.0 - 223.255.255.255 default subnet mask: 255.255.255.0 (or /24, 24 network bits and 8 host bits)
Class D: (Used for multicasting, not used for IP addressing)
224.0.0.0 - 239.255.255.255
Class E: (Unused)
240.0.0.0 - 255.255.255.255

Subnetting based on the number of networks:

Example A:

Say that our assigned network is 213.213.213.0 and we need to divide it into 5 networks.

Let's consult a table for our decimal to binary conversion to find what 5 is in binary. We work with 8 bits and we start from 0, so we work from 20 to 27:

27 26 25 24 23 22 21 20
128 64 32 16 8 4 2 1
0 0 0 0 0 1 0 1

That means we need 3 bits to represent number 5 (22 is the 3rd bit, we start from 0). That also means we need to borrow 3 host bits for our network.

That means that in reality we will have 23=8 networks.

Class C subnet mask: 255.255.255.0 (/24) or in binary:   11111111 11111111 11111111 00000000
Our subnet mask will be the default mask with the number of host bits that we are required to borrow. In our case, we need to borrow 3 host bits and move them as network bits. Note that we always borrow the first available network bits, so:                  

Our  subnet mask:      11111111 11111111 11111111 11100000
In other words     :           255         255         255           224   (/27, since we have 27 network bits enabled)

So now, the number of our hosts will be the remaining host bits. In our case 32.
The number of usable hosts in our network though will be the remaining host bits minus the network address and the broadcast address, so 2 to the power of our remaining host bits minus 2.

So, in our case (25)-2=32-2=30. We will have 30 hosts in 8 networks.

A faster way to find this number is with what Cisco refers to as the interesting part. The interesting part is
the part of the subnet mask which is not 255. We subtract this number from 256 and that will be our IP address increment.

This in our case is the fourth part of the IP address. So, In our case, 256-224=32.

Another way to find our IP address increment is to find out the least number where "1" was put in our subnet mask (in our case that was the third place from the end) which corresponds to "32" if we do a lookup on our binary table. 

So our first IP allocation will be 213.213.213.32. Just write the increments and fill the spaces in-between.

213.213.213.0
213.213.213.32
213.213.213.64
213.213.213.96
etc.

So our network is:

213.213.213.0   - 213.213.213.31
213.213.213.32 - 213.213.213.63
213.213.213.64 - 213.213.213.95
etc.
.....

Example B:

We are given the network 130.130.0.0 and we want to divide it into 900 networks.

The sum of our 8-bit table (128 to 1) amounts to 255 (obviously). 8 bits are not enough. We need more entries:

1024 512 .................
0 1 .................

No need to calculate any more, we are sure that this is the correct bit, we go over after that.
So we need to borrow 10 host bits. 210=1024 networks.

This is a Class B network so its network mask looks like this:

11111111 11111111 00000000 00000000

So borrowing 10 host bits, our netmask looks like this:

11111111 11111111 11111111 11000000 (/26)
   255           255          255          192

256-192=64. So we have 64 hosts per network, 62 usable ones and our IP allocation
increments go like so:

130.130.130.0
130.130.130.64
130.130.130.128
etc.

Our IP ranges are:

130.130.130.0   - 130.130.130.63
130.130.130.64 - 130.130.130.127
etc.
.....

Example C:

We have the network 10.0.0.0 and we want to split it into 500 networks.

Let's consult a table for our decimal to binary conversion to find how 500 is represented:

The sum of our 8-bit table (128 to 1) amounts to 255 (obviously). 8 bits are not enough. We need one more entry:

256 128 64 32 16 8 4 2 1
1 1 1 1 1 0 1 0 0

That means that we need to borrow 9 host bits.

This is a class A address and its network mask looks like this:

11111111 00000000 00000000 00000000

So adding 9 bits makes our netmask like so:

11111111 11111111 10000000 00000000 (/17) or:
   255            255          128             0

29=512, so we have 512 networks.
256-128=128, so we have 128 hosts in total and 126 usable hosts per network,so our IP allocation increments and our ranges are the following:

10.0.0.0     - 10.0.127.255
10.0.128.0 - 10.0.255.255
10.1.0.0     - 10.1.127.255
10.1.128.0 - 10.1.255.255
10.2.0.0     - 10.2.127.255
10.2.128.0 - 10.2.255.255
.......

Now that we've gone through the basics and learned how to subnet, we need to note:
It's always a good idea to subtract one when trying to subnet based on networks just in case our number is on a binary calculation boundary (e.g. 128,64,32,16,8,4,2,1).

Let's go over an example again with our new method:

We have the network 192.168.0.0 and we need to divide it into 16 networks.

We need to subtract one so instead of calculating for 16, we calculate for 16-1=15:

128 64 32 16 8 4 2 1
0 0 0 0 1 1 1 1

We need 4 bits, so 24; that means we will have 16 networks.

Class C so:

11111111 11111111 11111111 11110000
   255           255          255          240           or /28 in CIDR notation.

And 24=16 IP addresses and 14 usable hosts. Our IP allocation is arranged by 256-240=16 increments.

192.168.0.0   - 192.168.0.15
192.168.0.16 - 192.168.0.31
192.168.0.32 - 192.168.0.63
etc.

If we had done the calculation without subtracting one, we would have provisioned for 32 networks.