Friday, January 31, 2014

Allow access to a host with a dynamic IP in MySQL

So I was working with the same client with the issue I described here and was setting up their MySQL.

Here's the thing though: When MySQL tries to resolve hostnames, its performance drops dramatically. What I would ideally like to do is keep the skip-name-resolve parameter in the MySQL configuration file, while avoiding allowing connections from everyone else to my client's database.

One way would of course be through iptables. One might say that since iptables allows only a specific set of addresses, then it would be OK to actually have a user like 'root'@'%'.

Honestly, it would probably be a secure scenario in most cases, but you never know. Someone might accidentally flush the iptables rules, allow access to all on port 3306, you get the picture. It's a client. And they're devs. And they'll have root access afterwards. The horror. The horror.

So what do we do? Simple. Let's change our script a bit.

Here we assume that the user we want to allow access to is root logging in from www.example.com with password "root_password". We also have a rule that allows root from 1.2.3.4 we want to keep. What we want is to check every so often if the IP address of www.example.com has changed, and if so, drop that user and create another one that will be able to connect from the new IP address.  

[root@server ~]# mysql -e 'select host,user from user' mysql
+----------------+----------+
| host           | user     |
+----------------+----------+
| 127.0.0.1      | usera    |
| 127.0.0.1      | root     |
| 127.0.0.1      | userb    |
| 1.2.3.4        | root     |
| 5.6.7.8        | root     |
+----------------+----------+

[root@server ~]# vi /root/nslookup.pl
#!/usr/bin/perl

use strict;
use warnings;
use DBI();

my $result_raw=`dig www.example.com`;
my @result=split(/\n/, $result_raw);
my $IP='0.0.0.0';
foreach my $line (@result)
{
   if ($line =~ /.*www.example.com.*/)
   {
      $IP = $line;
   }
}
chomp $IP;
$IP=~ s/(.*\s+)(\d.*)/$2/;
if ($IP =~ /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/ and $IP ne '0.0.0.0')
{
   system("/sbin/iptables -D INPUT 3");
   system("/sbin/iptables -I INPUT 3 -s $IP -j ACCEPT");
}
else
{
   exit 2;
}

my $users=`mysql -e 'select host,user from user' mysql`;
my $OLD_IP='0.0.0.0';
@result=split(/\n/, $users);
foreach my $line (@result)
{
     if ($line =~ /.*(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*root.*/)
     {
        if (($line =~ /.*localhost.*/) || ($line =~ /.*127\.0\.0\.1.*/) || ($line =~ /.*1\.2\.3\.4.*/))
        {
            #print "skipping\n";
        }
        else
        {
            $OLD_IP = $line;
        }
     }
}
$OLD_IP =~ s/^((\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))(.*)/$1/;
chomp $OLD_IP;
if ($OLD_IP =~ /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/ and $OLD_IP ne '0.0.0.0')
{
   if ($OLD_IP ne $IP)
   {
      my $dbh = DBI->connect("DBI:mysql:database=mysql;host=127.0.0.1", "root", "root_password", {'RaiseError' => 1});
      my $query="DROP USER 'root'\@'$OLD_IP'";
      $dbh->do($query);
      $query="CREATE USER 'root'\@'$IP' IDENTIFIED BY 'root_password'";
      $dbh->do($query);
      $query="GRANT ALL PRIVILEGES ON *.* TO 'root'\@'$IP'";
      $dbh->do($query);
      $dbh->disconnect();
   }
}
[root@server ~]# crontab -e
*/5 * * * * /usr/bin/perl /root/nslookup.pl > /dev/null 2>&1
With this script and using crontab we a) check every 5 minutes the IP of www.example.com; b) we replace the old iptables rule with a new one if needed; c) we scan through the users that are allowed to access MySQL, ignore everyone who is not root and addresses such as "localhost" (this shouldn't really exist if skip-name-resolve is on, but it's ok to have it in the script just in case), "127.0.0.1", and "1.2.3.4" (the rule that allows root from 1.2.3.4 we wanted to keep) and in case the IP has changed replace that user with a user from the new IP.

So if all goes to plan with our regular expressions and our pattern matching, the user 'root'@'5.6.7.8' will be dropped and a new one will be created:
[root@server ~]# mysql -e 'select host,user from user' mysql
+----------------+----------+
| host           | user     |
+----------------+----------+
| 127.0.0.1      | usera    |
| 127.0.0.1      | root     |
| 127.0.0.1      | userb    |
| 1.2.3.4        | root     |
| 9.10.11.12     | root     |
+----------------+----------+

Note that you'll need the perl-DBI and perl-DBD-MySQL packages in order to run this script.

Fun, right? Right?... Meh, I guess not.

Saturday, January 25, 2014

XenServer General Backend Error while trying to detach an NFS Storage Repository

Ever had this error? Really annoying, and you're stuck basically until the NFS server comes up (if you're lucky).

It's more or less practice for some people (like me) to have their O/S images uploaded to and made available from an EC2 instance so that whenever a new VM is about to be installed, that instance can be turned on and accessed by XenServer as an NFS ISO Library.


Here's the catch though: EC2 instances have dynamic IPs, so if you forget to detach the SR from XenServer before you shut the instance down it becomes inaccessible and therefore XenServer can't detach it gracefully.


Of course that is not the only case where this error can occur. If your NFS server bites the dust, you're stuck with a Storage Repository that just won't die. The workaround is pretty straightforward though.

- SSH to your XenServer
- Find out the UUID of your NFS Storage (xe sr-list name-label=YOUR_NFS_SR_NAME)
- Find out the SR-UUID of the corresponding Physical Block Device (xe pbd-list sr-uuid=YOUR_NFS_STORAGE_UUID)
- Find out the location of the filesystem where the zombie NFS export is mounted on
- Unmount it using the "force" and "lazy" parameters
- Order XenServer to unplug the Physical Block Device through the command line (xe pbd-unplug uuid=UUID_OF_THE_PBD --although now this should be possible from the GUI as well)
- Order XenServer to forget the device altogether (xe sr-forget uuid=YOUR_NFS_STORAGE_UUID --although now this should be possible from the GUI as well)

For our purposes, our XenServer's IP address is 172.16.1.10, the remote NFS share is located at 54.217.159.179:/export and our NFS SR was named "NFS ISO library".

root@linux:~# ssh -C 172.16.1.10   
[root@xenserver ~]# xe sr-list name-label="NFS ISO library" 
uuid ( RO)                : 6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d
          name-label ( RW): NFS ISO library
    name-description ( RW): NFS ISO Library [54.217.159.179:/export]
                host ( RO): xenserver
                type ( RO): iso
        content-type ( RO): iso


[root@xenserver ~]# xe pbd-list sr-uuid=6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d
uuid ( RO)                  : accefe2d-8a22-d87d-23c6-53ab3110fc6b
             host-uuid ( RO): 66807c1f-39c3-418f-8d91-301380a05805
               sr-uuid ( RO): 6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d
         device-config (MRO): type: nfs_iso; location: 54.217.159.179:/export
    currently-attached ( RO): true

[root@xenserver ~]# cat /etc/mtab
/dev/md0 / ext3 rw 0 0
none /proc proc rw 0 0
none /sys sysfs rw 0 0
none /dev/pts devpts rw 0 0
none /dev/shm tmpfs rw 0 0
/opt/xensource/packages/iso/XenCenter.iso /var/xen/xc-install iso9660 \
    ro,loop=/dev/loop0 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
54.217.159.179:/export /var/run/sr-mount/6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d \
    nfs rw,soft,timeo=133,retrans=2147483647,tcp,actimeo=0,addr=54.217.159.179 0 0

[root@xenserver ~]# umount -f -l /var/run/sr-mount/6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d
[root@xenserver ~]# xe pbd-unplug uuid=accefe2d-8a22-d87d-23c6-53ab3110fc6b
[root@xenserver ~]# xe sr-forget uuid=6e0b7373-a1a3-1edb-1a09-b72a2bebbc7d

Configure iptables to allow a dynamic IP

A client asked me to configure their servers' iptables and make them as secure as possible. The catch was that they wanted these servers accessible from their offices, which didn't have a static IP (of course).

Luckily enough, they did have an account with DynDNS. What I did was the following:

[root@server ~]# iptables -F INPUT  
[root@server ~]# iptables -A INPUT -i lo -j ACCEPT   
[root@server ~]# iptables -A INPUT -s server.IP -j ACCEPT   
[root@server ~]# iptables -A INPUT -s server.IP -j ACCEPT   
....  
more iptables rules follow  
....  
[root@server ~]# iptables -F FORWARD  
[root@server ~]# iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited   
[root@server ~]# iptables -A OUTPUT -j ACCEPT  
[root@server ~]# iptables-save > /etc/sysconfig/iptables  

So besides allowing everything coming from the loopback interface, which is always the first iptables rule, I entered a duplicate entry.

If you're wondering then let me tell you that there's a method to my madness. Now I'm going to write a script that resolves my client's IP address using their DynDNS hostname and replaces that duplicate line with that address. 

[root@server ~]# yum install bind-utils  
[root@server ~]# vi /root/nslookup.pl
#!/usr/bin/perl  
   
use strict;  
use warnings;  
   
my $result_raw=`dig www.example.com`;
my @result=split(/\n/, $result_raw);
my $IP='0.0.0.0';  
foreach my $line (@result)  
      {  
           if ($line =~ /.*www.example.com.*/)  
                {  
                     $IP = $line;       
                }  
      }   
chomp $IP;  
$IP=~ s/(.*\s+)(\d.*)/$2/;  
if ($IP =~ /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/ and $IP ne '0.0.0.0')  
      {  
           system("/sbin/iptables -D INPUT 3");  
           system("/sbin/iptables -I INPUT 3 -s $IP -j ACCEPT");  
      }  
[root@server ~]# crontab -e
*/5 * * * * /usr/bin/perl /root/nslookup.pl > /dev/null 2>&1

This process is for Red Hat based systems. For Debian-based systems the process is slightly different:

root@server ~# iptables -F INPUT   
root@server ~# iptables -A INPUT -i lo -j ACCEPT    
root@server ~# iptables -A INPUT -s server.IP -j ACCEPT    
root@server ~# iptables -A INPUT -s server.IP -j ACCEPT    
....   
more iptables rules follow   
....   
root@server ~# iptables -F FORWARD   
root@server ~# iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited    
root@server ~# iptables -A OUTPUT -j ACCEPT   
root@server:~# iptables-save > /etc/iptables.up.rules  
   
root@server:~# vi /etc/init.d/iptables_fw
#!/bin/sh  
### BEGIN INIT INFO  
# Provides:     iptables_init  
# Required-Start:  $local_fs $network  
# Required-Stop:  
# Default-Start:   2 3 4 5       
# Default-Stop:   0 1 6  
# Short-Description: Firewall script  
# Description:    Start iptables-based firewall  
### END INIT INFO  
#  
iptables-restore < /etc/iptables.up.rules  

root@server:~# chmod 755 /etc/init.d/iptables_fw  
root@server:~# service iptables_fw start  
root@server:~# update-rc.d iptables_fw defaults
root@server:~# vi /root/nslookup.pl
#!/usr/bin/perl  
  
use strict;  
use warnings;  
   
my $result_raw=`dig www.example.com`;
my @result=split(/\n/, $result_raw);
my $IP='0.0.0.0';  
foreach my $line (@result)  
      {  
           if ($line =~ /.*www.example.com.*/)  
                {  
                     $IP = $line;       
                }  
      }   
chomp $IP;  
$IP=~ s/(.*\s+)(\d.*)/$2/;  
if ($IP =~ /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/ and $IP ne '0.0.0.0')  
      {  
           system("/sbin/iptables -D INPUT 3");  
           system("/sbin/iptables -I INPUT 3 -s $IP -j ACCEPT");  
      }  
root@server:~# crontab -e
*/5 * * * * /usr/bin/perl /root/nslookup.pl > /dev/null 2>&1