Wednesday, July 9, 2014

Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 1

I'm a bit partial to Zen Load Balancer. As a matter of a fact, I love it. It has many, many things ready to go from the start.

And it being just a Debian distro with the zenloadbalancer package on top, there's a lot you can do to customize it. The first thing we need to do is get rid of this:



My system has 16GB of memory but memory reported is just 3GB? Yup.
See Zen Load Balancer is a 32-bit app and it is distributed with a 32-bit Debian distro.

Assuming we have a 64-bit system with more memory installed we'll need to upgrade the kernel to a PAE one. This is both a performance and a security enhancement. It will allow us to use more memory and will also enable NX protection (provided that our BIOS and CPU support it too), as the NX bit works on the 63rd bit of the address.

Editing our repos first:
root@zen-lb:~# vi /etc/apt/sources.list
#official repository for Debian
deb http://ftp.debian.org/debian/ stable main non-free
deb-src http://ftp.debian.org/debian/ stable main non-free
deb http://security.debian.org/ stable/updates main
deb-src http://security.debian.org/ stable/updates main
#official repository for Zen Load Balancer Updates
deb http://zenloadbalancer.sourceforge.net/apt/x86 v3/

#Let's add this repo as well to do a moderate PAE upgrade at first
deb http://security.debian.org/debian-security squeeze/updates main

Let's try to upgrade our kernel now:
root@zen-lb:~# apt-get update
....
....
....
Reading package lists... Done
W: GPG error: http://ftp.debian.org stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B48AD6246925553 NO_PUBKEY 6FB2A1C265FFB764
W: GPG error: http://security.debian.org stable/updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B48AD6246925553

Yeah ok, classic Debian thing:
root@zen-lb:~# gpg --keyserver pgpkeys.mit.edu --recv-key 6FB2A1C265FFB764
root@zen-lb:~# gpg -a --export 6FB2A1C265FFB764 | apt-key add -
root@zen-lb:~# gpg --keyserver pgpkeys.mit.edu --recv-key 8B48AD6246925553
root@zen-lb:~# gpg -a --export 8B48AD6246925553 | apt-key add -
root@zen-lb:~# apt-get update
Fetched 548 kB in 1s (444 kB/s)
Reading package lists... Done

One more time for the world:
root@zen-lb:~# apt-cache search linux-image
linux-image-2.6-486 - Linux for older PCs (dummy package)
linux-image-2.6-686 - Linux for modern PCs (dummy package)
linux-image-2.6-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
linux-image-2.6-686-pae - Linux for modern PCs (dummy package)
linux-image-2.6-amd64 - Linux for 64-bit PCs (dummy package)
linux-image-486 - Linux for older PCs (meta-package)
linux-image-686 - Linux for modern PCs (dummy package)
linux-image-686-bigmem - Linux for PCs with 4GB+ RAM (dummy package)
linux-image-686-pae - Linux for modern PCs (meta-package)
linux-image-amd64 - Linux for 64-bit PCs (meta-package)
linux-image-rt-686-pae - Linux for modern PCs (meta-package), PREEMPT_RT
linux-image-3.2.0-4-486 - Linux 3.2 for older PCs
linux-image-3.2.0-4-686-pae - Linux 3.2 for modern PCs
linux-image-3.2.0-4-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-686-pae
linux-image-3.2.0-4-amd64 - Linux 3.2 for 64-bit PCs
linux-image-3.2.0-4-rt-686-pae - Linux 3.2 for modern PCs, PREEMPT_RT
linux-image-3.2.0-4-rt-686-pae-dbg - Debugging symbols for Linux 3.2.0-4-rt-686-pae
linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486
linux-headers-2.6.32-5-686 - Header files for Linux 2.6.32-5-686
linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem
linux-headers-2.6.32-5-amd64 - Header files for Linux 2.6.32-5-amd64
linux-headers-2.6.32-5-openvz-686 - Header files for Linux 2.6.32-5-openvz-686
linux-headers-2.6.32-5-vserver-686 - Header files for Linux 2.6.32-5-vserver-686
linux-headers-2.6.32-5-vserver-686-bigmem - Header files for Linux 2.6.32-5-vserver-686-bigmem
linux-headers-2.6.32-5-xen-686 - Header files for Linux 2.6.32-5-xen-686
linux-image-2.6.32-5-486 - Linux 2.6.32 for old PCs
linux-image-2.6.32-5-686 - Linux 2.6.32 for modern PCs
linux-image-2.6.32-5-686-bigmem - Linux 2.6.32 for PCs with 4GB+ RAM
linux-image-2.6.32-5-686-bigmem-dbg - Debugging infos for Linux 2.6.32-5-686-bigmem
linux-image-2.6.32-5-amd64 - Linux 2.6.32 for 64-bit PCs
linux-image-2.6.32-5-openvz-686 - Linux 2.6.32 for modern PCs, OpenVZ support
linux-image-2.6.32-5-openvz-686-dbg - Debugging infos for Linux 2.6.32-5-openvz-686
linux-image-2.6.32-5-vserver-686 - Linux 2.6.32 for modern PCs, Linux-VServer support
linux-image-2.6.32-5-vserver-686-bigmem - Linux 2.6.32 for PCs with 4GB+ RAM, Linux-VServer support
linux-image-2.6.32-5-vserver-686-bigmem-dbg - Debugging infos for Linux 2.6.32-5-vserver-686-bigmem
linux-image-2.6.32-5-xen-686 - Linux 2.6.32 for modern PCs, Xen dom0 support
linux-image-2.6.32-5-xen-686-dbg - Debugging infos for Linux 2.6.32-5-xen-686

Right, let's be conservative and upgrade to a 2.6 PAE kernel, we'll do a major upgrade later:
root@zen-lb:~# uname -a
Linux zen-lb 2.6.32-5-686 #1 SMP Wed Jan 12 04:01:41 UTC 2011 i686 GNU/Linux
root@zen-lb:~# apt-get install linux-image-2.6.32-5-686-bigmem
Get:1 http://ftp.debian.org/debian/ stable/main linux-base all 3.5 [34.3 kB]
Get:2 http://security.debian.org/debian-security/ squeeze/updates/main linux-image-2.6.32-5-686-bigmem i386 2.6.32-48squeeze6 [27.6 MB]
Get:3 http://ftp.debian.org/debian/ stable/main firmware-linux-free all 3.2 [20.7 kB]
Fetched 27.7 MB in 20s (1,366 kB/s)
Preconfiguring packages ...
(Reading database ... 18065 files and directories currently installed.)
Preparing to replace linux-base 2.6.32-30 (using .../linux-base_3.5_all.deb) ...
Unpacking replacement linux-base ...
Selecting previously deselected package linux-image-2.6.32-5-686-bigmem.
Unpacking linux-image-2.6.32-5-686-bigmem (from .../linux-image-2.6.32-5-686-bigmem_2.6.32-48squeeze6_i386.deb) ...
Selecting previously deselected package firmware-linux-free.
Unpacking firmware-linux-free (from .../firmware-linux-free_3.2_all.deb) ...
Processing triggers for man-db ...
Setting up linux-base (3.5) ...
Setting up linux-image-2.6.32-5-686-bigmem (2.6.32-48squeeze6) ...
Running depmod.
Running update-initramfs.
update-initramfs: Generating /boot/initrd.img-2.6.32-5-686-bigmem
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 2.6.32-5-686-bigmem /boot/vmlinuz-2.6.32-5-686-bigmem
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 2.6.32-5-686-bigmem /boot/vmlinuz-2.6.32-5-686-bigmem
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-5-686-bigmem
Found initrd image: /boot/initrd.img-2.6.32-5-686-bigmem
Found linux image: /boot/vmlinuz-2.6.32-5-686
Found initrd image: /boot/initrd.img-2.6.32-5-686
done
Setting up firmware-linux-free (3.2) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-2.6.32-5-686-bigmem
root@zen-lb:~# reboot

All right, so did it work?

Looks good. And what about NX?
root@zen-lb:~# dmesg | grep ".*NX.*protection"
[    0.000000] NX (Execute Disable) protection: active

Cool. Let's go on then. Let's do a distro upgrade from Squeeze to Wheezy:
root@zen-lb:~# apt-get dist-upgrade
root@zen-lb:~# reboot

Indeed:
root@zen-lb:~# cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"
NAME="Debian GNU/Linux"
VERSION_ID="7"
VERSION="7 (wheezy)"
ID=debian
ANSI_COLOR="1;31"
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support/"
BUG_REPORT_URL="http://bugs.debian.org/"
root@zen-lb:~# cat /proc/version
Linux version 3.2.0-4-686-pae (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.60-1+deb7u1

The first thing I'm going to do is tune my filesystem. To be honest, this is just a load balancer so I can afford to lose a few seconds of logs if the power goes down.
root@zen-lb:~# vi /etc/fstab
proc            /proc           proc    defaults        0       0
# / was on /dev/sdb3 during installation
UUID=b6016824-536e-43bc-8f1f-fbfd2fab146d /               ext4    noatime,nodiratime,nobarrier,nobh,commit=120,data=writeback,journal_async_commit,errors=remount-ro 0       1
# /boot was on /dev/sdb1 during installation
UUID=6d4bd9ca-ba29-4700-b90c-07c614d79f0e /boot           ext4    defaults        0       2
# swap was on /dev/sdb2 during installation
UUID=3cb4eb4d-b0ab-4b60-825b-fc0224356580 none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

Go to single user mode and tune my root filesystem (mine is on /dev/sdb3):
root@zen-lb:~# init 1
root@zen-lb:~# tune2fs -O dir_index /dev/sdb3
root@zen-lb:~# umount -a
root@zen-lb:~# e2fsck -D /dev/sdb3

And now increase the number of open files limit:
root@zen-lb:~# vi /etc/security/limits.conf
....
* soft nofile 65536
* hard nofile 65536
# End of file

When regular users log in, they get an open files warning, let's correct it by uncommenting these few lines of code in /etc/profile:
root@zen-lb:~# vi /etc/profile
....
#if [ -f /etc/sysctl.conf ]; then
# FILEMAX=`grep "^fs.file-max.*=" /etc/sysctl.conf | awk -F'=' '{printf $2}'`
# if [ "$FILEMAX" != "" ]; then
#  ulimit -n $FILEMAX
# fi
#fi

Finally, let's update our repos to correctly receive wheezy updates:
root@zen-lb:~# vi /etc/apt/sources.list
#official repository for Debian
deb http://ftp.debian.org/debian wheezy main contrib non-free
deb-src http://ftp.debian.org/debian wheezy main contrib non-free
deb http://ftp.debian.org/debian wheezy-updates main contrib non-free

deb http://http.debian.net/debian wheezy main contrib non-free
deb-src http://http.debian.net/debian wheezy main contrib non-free

deb http://http.debian.net/debian wheezy-updates main contrib non-free
deb-src http://http.debian.net/debian wheezy-updates main contrib non-free

deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free

#official repository for Zen Load Balancer Updates
deb http://zenloadbalancer.sourceforge.net/apt/x86 v3/

All right, I guess we didn't do that much, but it's enough to call the end of Part 1.

Monday, June 23, 2014

XenServer 6.2 Unable to find partition containing kernel

Sometimes XenServer can be a real pain in the neck.

Here's what I ran into a little earlier:

I upgrade my VM's kernel and reboot. Wait for a while, but the VM doesn't appear to come up. So I go to my Logs and lo and behold:



Ugh, wat?

OK, let's see if we can start it manually, worst case scenario is the command line providing us with more info, right?

[root@xen]# xe vm-start vm="My VM's Name"
The bootloader returned an error
vm: .... (My VM's Name)
msg: Unable to find partition containing kernel

Oh, how informative. Luckily XenServer provides us with an off-line VM boot editor. That should do the trick:

[root@xen]# xe-edit-bootloader -n "My VM's Name" -p 1
Plugging VBD: 
Creating dom0 VBD: ...
add map ...1 (252:5): 0 1048576 linear /dev/sm/backend/.../... 2048
add map ...2 (252:6): 0 33554432 linear /dev/sm/backend/.../... 1050624
add map ...3 (252:7): 0 1886386176 linear /dev/sm/backend/.../... 34605056
Waiting for /dev/mapper/...1: .....Device /dev/mapper/...1 not found.
You must specify the correct partition number with -p
Unplugging VBD: . done

This should have worked. It's usually as good as logging into the VM and editing the GRUB menu.lst yourself. What on earth happened here? Well, I'm pretty sure I know my systems and my partitions so I'm sure the /boot partition is the first one!

What now? Well, for sure we'll need to boot from a rescue disk. So, make sure your NFS ISO library is online or you have a storage already handy that has a rescue CD .iso. No? We don't? Well, no reason to panic. We'll just create a directory named "/rescue/ISOs", download a Debian Live CD there and create an SR with the name label "RESCUE" that points to that directory:

[root@xen]# mkdir -p /rescue/ISOs
[root@xen]# cd /rescue/ISOs
[root@xen]# wget http://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/debian-live-7.5.0-amd64-rescue.iso
[root@xen]# xe sr-create name-label=RESCUE type=iso device-config:legacy_mode=true device-config:location=/rescue/ISOs content-type=iso
303bd588-c675-2bc0-8982-8a691141226a 

Please note that you need to make sure you create this SR on a partition with plenty of disk space, the Dom0's default partition is just too small and therefore I do not recommend having any rescue ISOs on it.

Cool. Now let's select that ISO as our DVD disk on our VM and finally boot off of it. The latter task can be done by going to VM -> Start/Shut Down -> Start in Recovery Mode.

Now that everything's started up, let's see what's going on. As I already mentioned, my /boot partition is my first partition so let's mount it and inspect the grub.conf file:

root@debian:~# cat /proc/partitions
major minor  #blocks  name

 202        0  960495616 xvda
 202        1     524288 xvda1
 202        2   16777216 xvda2
 202        3  943193088 xvda3
  11        0     696320 sr0
   7        0     548992 loop0 
root@debian:~# mkdir /a
root@debian:~# mount /dev/xvda1 /a
root@debian:~# vi /a/grub/grub.conf 
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/xvda3
#          initrd /initrd-[generic-]version.img
#boot=/dev/xvda
default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Oracle Linux Server (3.8.13-35.1.2.el6uek.x86_64)
        root (hd0,0)
        kernel /vmlinuz-3.8.13-35.1.2.el6uek.x86_64 ro root=UUID=e9b3edd9-15e0-4cfd-a8fa-7dc24f6aeefa rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD console=hvc0  KEYTABLE=us SYSFONT=latarcyrheb-sun16  rd_NO_LVM rd_NO_DM rhgb quiet
        initrd /initramfs-3.8.13-35.1.2.el6uek.x86_64.img

Now, we have two options really:

a) Solving this through XenServer. We would have to issue:

xe vm-param-set uuid=vm-uuid PV-bootloader-args="--kernel=/vmlinuz-3.8.13-35.1.2.el6uek.x86_64 --ramdisk=/initramfs-3.8.13-35.1.2.el6uek.x86_64.img"
xe vm-param-set uuid=vm-uuid PV-args="root=root-device  ro quiet"

Since I would have to change this every time I upgrade the kernel and I really want to find what went wrong, I'll pass this option for now.

b) Trying to debug what's wrong with grub.conf.

Since the kernel is too recent to not support XenServer, it should just be a matter of patience to debug it.

Here's what's usually wrong with grub.conf/menu.lst:
i  ) The root(hdx,y) is wrong:
          x should point to the hard drive number where our boot partition is located at;
          y should point to the partition number of our boot partition.
   In this case, root(hd0,0) is correct.
ii ) The paths of vmlinuz-... and/or initramfs-... are wrong.
         The paths should be relative to the partition root directory. So for example if the boot directory is in a dedicated partition, it should be /vmlinuz-... and /initramfs-... but if the boot directory is in the same partition as the linux root (/) directory it should be /boot/vmlinuz-... and /boot/initramfs... if that explanation makes sense.
iii) The root directive is wrong.
          Here I mean the root directive that defines the linux root (/) directory, and not the boot partition which has been already declared with the root(hdx,y) statement. It could be root=/dev/xvda3 for instance. In my case it is root=UUID=e9b3edd9-15e0-4cfd-a8fa-7dc24f6aeefa.

The three cases above can be easily examined with a simple ls command on the /a directory and a blkid /dev/xvda3 or ls -l /dev/disk/by-uuid to find if the UUID of the device that hosts our root (/) directory is the correct one.

Another case is for the partition to have been corrupted, so we just umount /a and fsck /dev/xvda1    

In my case, as you can see the error was in the default setting. Changing this to 0 did the trick. Default signifies which item in the menu is the one that will boot after the user-interaction timeout occurs. The count starts from 0 and not from 1 so my system couldn't boot. What happened is that the OS provider had decided to change the default kernel order, making a mess out of my server.

Save changes, shutdown, reboot. Should be fine.

Friday, June 6, 2014

Subnetting made easy Part 2

Class A:
0.0.0.0     - 127.255.255.255 default subnet mask: 255.0.0.0 (or /8, 8 network bits and 24 host bits)
Class B:
128.0.0.0 - 191.255.255.255 default subnet mask: 255.255.0.0 (or /16, 16 network bits and 16 host bits)
Class C:
192.0.0.0 - 223.255.255.255 default subnet mask: 255.255.255.0 (or /24, 24 network bits and 8 host bits)
Class D: (Used for multicasting, not used for IP addressing)
224.0.0.0 - 239.255.255.255
Class E: (Unused)
240.0.0.0 - 255.255.255.255

Subnetting based on the number of hosts:

Example A:

Say that our assigned network is 213.213.213.0 and we need to divide it into networks that have 30 hosts each.

Let's consult a table for our decimal to binary conversion to find what 30 is in binary. We work with 8 bits and we start from 0, so we work from 20 to 27:

2726252423222120
1286432168421
00011110

That means we need 5 bits to represent number 30 (24 is the 5th bit, we start from 0) and therefore we need 5 bits for our hosts. 52=32, and the number of the actual usable hosts is 52-2=30.

213.213.213.0 is a Class C network.

Class C subnet mask: 255.255.255.0 (/24) or in binary:   11111111 11111111 11111111 00000000

Our subnet mask will be the default mask with the number of network bits that we will need to have in order to have the number of hosts we desire.

Class C subnet mask: 11111111 11111111 11111111 00000000
                                                                                  ^^^ we need 5 host bits, so we'll need to turn the last 5 bits to 0; the rest will be assigned as network bits by turning them to 1.                    

Our     subnet mask: 11111111 11111111 11111111 11100000
 In other words     :      255          255       255           224   (/27, since we have 27 network bits enabled)

So now, the number of our hosts will be the remaining host bits. In our case 32. The network bits are 3, so 23=8 networks.

The number of usable hosts in our network though will be the remaining host bits minus the network
address and the broadcast address, so 2 to the power of our remaining host bits minus 2.

So, in our case 25-2=32-2=30. We will have 30 hosts -exactly as many as we needed- in 8 networks.

A faster way to find this number is with what Cisco refers to as the interesting part. The interesting part is
the part of the subnet mask which is not 255. We subtract this number from 256 and that will be our IP address increment.

This in our case is the fourth part of the IP address. Also, In our case, 256-224=32.

So our first IP allocation will be 213.213.213.32. Just write the increments and fill the spaces in-between.

213.213.213.0
213.213.213.32
213.213.213.64
213.213.213.96
etc.
So our network is:

213.213.213.0 - 213.213.213.31
213.213.213.32 - 213.213.213.63
213.213.213.64 - 213.213.213.95
etc.
.....

Example B:

We are given the network 130.130.0.0 and we want to divide it into networks with 600 hosts each.

Let's consult a table for our decimal to binary conversion to find how 600 is represented:

The sum of our 8-bit table (128 to 1) amounts to 255 (obviously). 8 bits are not enough. We need more entries:

5122561286432168421
1001011000


So we need 10 bits for our hosts. 102-2=1024 number of hosts, 1022 usable hosts.

This is a Class B network so its network mask looks like this:

11111111 11111111 00000000 00000000


We need 10 host bits, so we'll need to turn the last 10 bits to 0; the rest will be assigned as network bits by turning them to 1. And now our netmask looks like this:

11111111 11111111 11111100 00000000 (/22)
   255          255           252          0

As we already said, we need 10 bits for our hosts. 102-2=1024 number of hosts, 1022 usable hosts.
The network bits are 6, so 26=64 networks.

256-252=4. So this is our network:

130.130.0.0
130.130.4.0
130.130.8.0
etc.

and our IP ranges are:

130.130.0.0 - 130.130.3.255
130.130.4.0 - 130.130.7.255
etc.
.....

Example C:

We have the network 10.0.0.0 and we want to split it into networks with 96 hosts each.

Let's consult a table for our decimal to binary conversion to find how 96 is represented:

12864.................
01.................

No need to calculate any more, we are sure that this is the correct bit, we go over after that
So we need 7 bits for our hosts. 27=128 hosts, 128-2=126 usable addresses.

This is a Class A network so its netmask looks like ths:

11111111 00000000 00000000 00000000

We need to have 7 host bits, so our netmask should look like this:

11111111 11111111 11111111 10000000 (/25)
   255           255         255         128

So 27=128 hosts, 128-2=126 usable addresses, 217=131072 networks.

256-128=128. Our IP ranges are:

10.0.0.0 - 10.0.0.127
10.0.0.128 - 10.0.0.255
10.0.1.0 - 10.0.1.127
etc.

Now that we've gone through the basics and learned how to subnet, we need to note:
It's always a good idea to add one when trying to subnet based on hosts just in case our number is on a binary calculation boundary (e.g. 128,64,32,16,8,4,2,1).

Let's go over an example again with our new method:

Say that our assigned network is 213.213.213.0 and we need to divide it into networks that have 32 hosts each.

We need to add one so instead of calculating for 32, we calculate for 32+1=33:

Let's consult a table for our decimal to binary conversion to find what 33 is in binary.

1286432168421
00100001

We need 6 bits to represent number 33. 62=36, and the number of the actual usable hosts is 62-2=34.

213.213.213.0 is a Class C network.

Class C subnet mask: 255.255.255.0 (/24) or in binary:   11111111 11111111 11111111 00000000

Our subnet mask will be the default mask with the number of network bits that we will need to have in order to have the number of hosts we desire.

Class C subnet mask: 11111111 11111111 11111111 00000000
                                                                                                     
Our subnet mask:       11111111 11111111 11111111 11000000
In other words:             255          255          255         192   (/26, since we have 26 network bits enabled)

The network bits are 2, so 22=4 networks.

The number of usable hosts in our network will be the remaining host bits minus the network address and the broadcast address, so 2 to the power of our remaining host bits minus 2. As calculated earlier, 62=36, and the number of the actual usable hosts is 62-2=34.


This in our case is the fourth part of the IP address. Also, In our case, 256-192=64.

So our first IP allocation will be 213.213.213.64. Just write the increments and fill the spaces in-between.

213.213.213.0
213.213.213.64
213.213.213.128
213.213.213.192

So our network is this:

213.213.213.0 - 213.213.213.63
213.213.213.64 - 213.213.213.127
213.213.213.128 - 213.213.213.191
213.213.213.192 - 213.213.213.255

If we had done the calculation without adding one, we would have provisioned for 30 hosts (5 bits,25=32,30 usable hosts)

Sunday, May 25, 2014

Subnetting made easy Part 1

Class A:
0.0.0.0     - 127.255.255.255 default subnet mask: 255.0.0.0 (or /8, 8 network bits and 24 host bits)
Class B:
128.0.0.0 - 191.255.255.255 default subnet mask: 255.255.0.0 (or /16, 16 network bits and 16 host bits)
Class C:
192.0.0.0 - 223.255.255.255 default subnet mask: 255.255.255.0 (or /24, 24 network bits and 8 host bits)
Class D: (Used for multicasting, not used for IP addressing)
224.0.0.0 - 239.255.255.255
Class E: (Unused)
240.0.0.0 - 255.255.255.255

Subnetting based on the number of networks:

Example A:

Say that our assigned network is 213.213.213.0 and we need to divide it into 5 networks.

Let's consult a table for our decimal to binary conversion to find what 5 is in binary. We work with 8 bits and we start from 0, so we work from 20 to 27:

27 26 25 24 23 22 21 20
128 64 32 16 8 4 2 1
0 0 0 0 0 1 0 1

That means we need 3 bits to represent number 5 (22 is the 3rd bit, we start from 0). That also means we need to borrow 3 host bits for our network.

That means that in reality we will have 23=8 networks.

Class C subnet mask: 255.255.255.0 (/24) or in binary:   11111111 11111111 11111111 00000000
Our subnet mask will be the default mask with the number of host bits that we are required to borrow. In our case, we need to borrow 3 host bits and move them as network bits. Note that we always borrow the first available network bits, so:                  

Our  subnet mask:      11111111 11111111 11111111 11100000
In other words     :           255         255         255           224   (/27, since we have 27 network bits enabled)

So now, the number of our hosts will be the remaining host bits. In our case 32.
The number of usable hosts in our network though will be the remaining host bits minus the network address and the broadcast address, so 2 to the power of our remaining host bits minus 2.

So, in our case (25)-2=32-2=30. We will have 30 hosts in 8 networks.

A faster way to find this number is with what Cisco refers to as the interesting part. The interesting part is
the part of the subnet mask which is not 255. We subtract this number from 256 and that will be our IP address increment.

This in our case is the fourth part of the IP address. So, In our case, 256-224=32.

Another way to find our IP address increment is to find out the least number where "1" was put in our subnet mask (in our case that was the third place from the end) which corresponds to "32" if we do a lookup on our binary table. 

So our first IP allocation will be 213.213.213.32. Just write the increments and fill the spaces in-between.

213.213.213.0
213.213.213.32
213.213.213.64
213.213.213.96
etc.

So our network is:

213.213.213.0   - 213.213.213.31
213.213.213.32 - 213.213.213.63
213.213.213.64 - 213.213.213.95
etc.
.....

Example B:

We are given the network 130.130.0.0 and we want to divide it into 900 networks.

The sum of our 8-bit table (128 to 1) amounts to 255 (obviously). 8 bits are not enough. We need more entries:

1024 512 .................
0 1 .................

No need to calculate any more, we are sure that this is the correct bit, we go over after that.
So we need to borrow 10 host bits. 210=1024 networks.

This is a Class B network so its network mask looks like this:

11111111 11111111 00000000 00000000

So borrowing 10 host bits, our netmask looks like this:

11111111 11111111 11111111 11000000 (/26)
   255           255          255          192

256-192=64. So we have 64 hosts per network, 62 usable ones and our IP allocation
increments go like so:

130.130.130.0
130.130.130.64
130.130.130.128
etc.

Our IP ranges are:

130.130.130.0   - 130.130.130.63
130.130.130.64 - 130.130.130.127
etc.
.....

Example C:

We have the network 10.0.0.0 and we want to split it into 500 networks.

Let's consult a table for our decimal to binary conversion to find how 500 is represented:

The sum of our 8-bit table (128 to 1) amounts to 255 (obviously). 8 bits are not enough. We need one more entry:

256 128 64 32 16 8 4 2 1
1 1 1 1 1 0 1 0 0

That means that we need to borrow 9 host bits.

This is a class A address and its network mask looks like this:

11111111 00000000 00000000 00000000

So adding 9 bits makes our netmask like so:

11111111 11111111 10000000 00000000 (/17) or:
   255            255          128             0

29=512, so we have 512 networks.
256-128=128, so we have 128 hosts in total and 126 usable hosts per network,so our IP allocation increments and our ranges are the following:

10.0.0.0     - 10.0.127.255
10.0.128.0 - 10.0.255.255
10.1.0.0     - 10.1.127.255
10.1.128.0 - 10.1.255.255
10.2.0.0     - 10.2.127.255
10.2.128.0 - 10.2.255.255
.......

Now that we've gone through the basics and learned how to subnet, we need to note:
It's always a good idea to subtract one when trying to subnet based on networks just in case our number is on a binary calculation boundary (e.g. 128,64,32,16,8,4,2,1).

Let's go over an example again with our new method:

We have the network 192.168.0.0 and we need to divide it into 16 networks.

We need to subtract one so instead of calculating for 16, we calculate for 16-1=15:

128 64 32 16 8 4 2 1
0 0 0 0 1 1 1 1

We need 4 bits, so 24; that means we will have 16 networks.

Class C so:

11111111 11111111 11111111 11110000
   255           255          255          240           or /28 in CIDR notation.

And 24=16 IP addresses and 14 usable hosts. Our IP allocation is arranged by 256-240=16 increments.

192.168.0.0   - 192.168.0.15
192.168.0.16 - 192.168.0.31
192.168.0.32 - 192.168.0.63
etc.

If we had done the calculation without subtracting one, we would have provisioned for 32 networks.

Tuesday, May 13, 2014

Using SSMTP

Let me quote from PCI-DSS questionaire D, for merchants:

1.2.1(a): Is inbound and outbound traffic restricted to that which is necessary for the cardholder data environment?
1.2.1(b): Is all other inbound and outbound traffic specifically denied (for example by using an explicit “deny all” or an implicit deny after allow statement)?

1.3.3: Are direct connections prohibited for inbound or outbound traffic between the Internet and the cardholder data environment?

10.5.5: Is file-integrity monitoring or change-detection software used on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert)?

10.6.1(a): Are written policies and procedures defined for reviewing the following at least daily, either manually or via log tools?
- All security events
- Logs of all system components that store, process, or transmit CHD and/or SAD, or that could impact the security of CHD and/or SAD
- Logs of all critical system components
- Logs of all servers and system components that perform security functions (for example, firewalls, intrusion-detection systems/intrusion-prevention systems (IDS/IPS), authentication servers, e-commerce redirection servers, etc.)
10.6.1(b) Are the above logs and security events reviewed at least daily?

So, having these prerequisites in mind we can assume that a few of the things that PCI-DSS requires us to do are:

a) Deny all direct connections from the internet to the cardholder environment.
b) Deny all internet traffic from our servers to the internet unless this traffic is a response to a valid request.
c) Install an IDS on our system.
d) Check our systems' log files every single day.


So something like the network above would fit the bill. An isolated cardholder environment, with our load balancer being our systems' default gateway. A few iptables rules here and there and unsolicited internet access is denied from and to our cardholder environment. Cool.

Wait a minute. What did you just say? Did you just tell me that I have to log in manually onto every single server in my network and check the log files? Since my servers can't be connected to the internet there's no other way to do this, right? I will have to spend my entire day SSHing from host to host, looking at log files. What a dream job. All right, here's the real problem, so let's try to find ways to solve it.

Ideally, you want to have your systems mail you reports. How are my network interfaces? Are they all up? How about my IDS? Have any critical files changed? How are my RAID arrays? etc.

If your systems email you reports instead of having to manually logging in and checking a bunch of log files, you're looking of spending about 15 minutes every morning instead of countless hours.

So the core of the problem is exactly this:

How do we go about sending emails from my systems that are not directly connected to the internet to myself?

Image taken from http://debian-handbook.info/browse/stable/network-services.html
Solution a): Install Postfix on every single system
Well, OK I guess. But we'll have to allow access to a DNS server; that means that we'll have to have direct access to that host. It's not exactly "the internet", it's just one (or maybe two) host(s) so I guess we can handle that. And then to our mail server. So for this solution to work we need:
i  ) Set up TCP forwarding between these IP addresses on our load balancer.
ii ) Set up exceptions on our iptables.
iii) Install and configure Postfix on all our servers.

Solution b): Install Postfix and a DNS server on the load balancer
This way we can lie to our systems. Essentially we'll be installing a DNS server only to create a split-brain DNS situation. That way, when our server needs to send an email they will use the mail server on our load balancer, which has access to the internet. We need to:
i  ) Install and configure something like BIND or PowerDNS on our load balancer.
ii ) Create our zones, forwarders, etc that will be used by our LAN and make sure that our MX record points to the internal IP of our load balancer (192.168.0.10 in my example).
iii) Install and configure Postfix on our load balancer.  

Solution c): Install Postfix on our load balancer and use SSMTP on our systems instead of the default mail client.
SSMTP is a mail client replacement that can lie. Awesome. We only need to do the following:
i ) Install and configure Postfix on our load balancer.
ii) Install and configure SSMTP on our systems.

So solution c is the most secure and uses the less resources in comparison to all other solutions. No need to guess which one I'm going to use! 

I'm going to be using a RHEL-based distro for our example.

SSMTP can be found in the EPEL repo so if we don't have it let's download and enable it.

This is for release 6.x :
[root@server ~]# wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@server ~]# rpm -Uvh epel-release-6*.rpm

In case you still have a 5.x release:
[root@server ~]# wget http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
[root@server ~]# rpm -Uvh epel-release-5*.rpm

Install SSMTP:
[root@server ~]# yum --enablerepo=epel -y install ssmtp

Time to configure SSMTP. The first file we need to configure is the revaliases file. This maps your users to mail server users. In my example I'll just map the local root account to the mail server postmaster account and the local www-data acount to the mail server no.reply account:
[root@server ~]# vi /etc/ssmtp/revaliases
# sSMTP aliases
#
# Format:       local_account:outgoing_address:mailhub
#
# Example: root:your_login@your.domain:mailhub.your.domain[:port]
# where [:port] is an optional port number that defaults to 25.

root:postmaster@foobar.com:192.168.0.10:25
www-data:no.reply@foobar.com:192.168.0.10:25

The other file we need to configure is the actual configuration file, the ssmtp.conf file. That's where you can specify if your fake mail server IP address, if it needs TLS, etc:
[root@server ~]# vi /etc/ssmtp/ssmtp.conf
#
# /etc/ssmtp.conf -- a config file for sSMTP sendmail.
#
# See the ssmtp.conf(5) man page for a more verbose explanation of the
# available options.
#

root=postmaster@foobar.com #That's where local root account email (such as system notifications) will be sent.
mailhub=192.168.0.10:25 #The IP address of our mailhub, in our case our load balancer.
#RewriteDomain=foobar.com #If you need to rewrite your mail headers so that the source of the IP address and the domain match use this option.
Hostname=smtp.foobar.com #If you need to rewrite the hostname of this system, use this option. domain (foobar.com) is optional.
FromLineOverride=YES #If you need to rewrite the "From:" header use this. Setting this to YES uses the From: value provided by the program that sends the e-mail. 
#UseTLS=YES #If you set up your mail server to use TLS, use this.
#AuthUser=no.reply@foobar.com #The username indicated here should be the username used to log into your mail server.
#AuthPass=password #The password indicated here should be the username used to log into your mail server.

# Use SSL/TLS certificate to authenticate against smtp host.
#UseTLSCert=YES

# Use this RSA certificate.
#TLSCert=/etc/pki/tls/private/ssmtp.pem

# Get enhanced (*really* enhanced) debugging information in the logs
# If you want to have debugging of the config file parsing, move this option
# to the top of the config file and uncomment
#Debug=YES

Now we need to specify that we want to use ssmtp as our MTA (if we have another MTA already installed):
[root@server ~]# alternatives --config mta
There are 2 programs which provide 'mta'.

  Selection    Command
-----------------------------------------------
*+ 1           /usr/sbin/sendmail.postfix
   2           /usr/sbin/sendmail.ssmtp

Enter to keep the current selection[+], or type selection number: 
2

Done!