Monday, December 15, 2014

Windows System error 1219, multiple connections to a server or shared resource by the same user

Here's a funny one I ran into earlier.

So I try to mount a SAMBA share on a Windows system. I get this error:



Wait, are you telling me that I am restricted to having only one share subdirectory connected per PC? What happens if I need two? Now that's some crappy engineering right there Batman!

Oh, I know what this is. It's probably one of those Windows "features" and if I do the same thing from the command line, it'll just work, right? Right? Wrong.

F:\Documents and Settings\user>net use Q: \\192.168.0.254\DIR2 /USER:shareuser /PERSISTENT:NO
The password is invalid for \\192.168.0.254\DIR2.

Enter the password for 'shareuser' to connect to '192.168.0.254':
Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed. Disconnect all previous connections to the server or shared resource and try again..

Through some digging I managed to find out that this is by design. Windows will connect to a share once; if you try to connect again, it will block you. If you want to connect to a different subdirectory of your SAMBA server, you will need to disconnect from the one you are currently connected to.

Fret not though, for this is Windows; home of the crappiest, nastiest pieces of engineering ever came to existence.

Workaround? Just edit your %WINDIR%/system32/drivers/etc/hosts file and add some more entries that correspond to your share's IP address. For instance:

192.168.0.254 foo
192.168.0.254 bar
192.168.0.254 foobar
192.168.0.254 fubar

And now I can connect to my share using my new aliases:

F:\Documents and Settings\user>net use Q: \\fubar\DIR2 /USER:shareuser /PERSISTENT:NO
The password is invalid for \\fubar\DIR2.

Enter the password for 'shareuser' to connect to 'fubar':
The command completed successfully.

Oh Windows how I hate thee with all my passion.

Friday, December 5, 2014

Recover BMC administrator password on HP Proliant DL180 G6

This is from my notes. I don't have an HP Proliant DL180 G6 server in front of me, but I trust my notes and I think this description should be as accurate as it can. You'll need a PC with an internet connection, a USB drive and a jumper connector. You might also need a Linux Live CD (the instructions here assume that you have a RHEL-based Live CD).

a) Disconnect AC power from the system
b) Insert jumper on connector J27, D group pins 
c) Update BMC firmware using HP ROMPaq Firmware Upgrade for HP ProLiant G6 Lights-Out 100 Remote Management (For USB Key-Media).
    Detailed information is available at the following Web page: http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=3884343&swItem=MTX-bd27c5aa4f134285aa4825e143&prodNameId=3884344&swEnvOID=1005&swLang=13&taskId=135&mode=4&idx=1
d) Copy the impitool.exe binary from http://www.intel.com/design/servers/ipmi/ipmi_tool.htm onto the USB drive
e) Remove AC power and remove the jumper from J27, D group pins.
f) After BMC recovery is complete, boot from the USB key again and do:
ipmitool.exe 20 18 47 03 02 61 64 6d 69 6e 00 00 00 00 00 00 00 00 00 00 00
g) Remove USB drive and shutdown.
h) If the admininstrator user was "admin" you can now log in as admin/admin. If you still can't log in, proceed as follows (the following instructions are for a RHEL Live CD):
  1. Get a Linux Live CD
  2. yum update
  3. modprobe ipmi_msghandler
  4. modprobe ipmi_devintf
  5. modprobe ipmi_si
  6. yum search ipmi
  7. yum -y install ipmitool
  8. ipmitool user list
  9. note down the name of user id #3 and log in with that as a username and password admin.

Wednesday, December 3, 2014

PCI-DSS, nginx access logs, and credit card masking

PCI-DSS states that the card number must not be displayed in full—no more than the first six and the last four digits may be visible on a screen, a receipt, or on any other media used by the organization.

Sometimes your web server might leave access logs that contain this sort of information though. Here's an nginx access log snippet:

1.2.3.4 - - [02/Dec/2014:14:58:54 +0200] "GET /ccsite/?card_number=5100+1500+0000+0001&amount=10000&cvv=123&holder_name=N.Ame&expiration_month=1&expiration_year=2015&description=Something&_method=POST HTTP/1.1" 200 395 "https://partnersite.com/apps/ccapp?dothis.js" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0"
1.2.3.4 - - [02/Dec/2014:16:07:46 +0200] "password=12345678&username=my_username" 200 416 "-" "Dalvik/1.6.0 (Linux; U; Android 4.3; C1905 Build/15.4.A.1.9)" 

Ouch, we shouldn't be writing stuff like this to our logs!

So what so we do?

Well, in this example I'm going to install the ngx_http_perl_module and use regular expressions to mask out the credit card and CVV.

First of all, let's see if we have Perl installed. Also, in order for Perl to recompile the modified modules during reconfiguration, it should be built with the -Dusemultiplicity=yes or -Dusethreads=yes parameters. Also, to make Perl leak less memory at run time, it should be built with the -Dusemymalloc=no parameter.

Let's check if we're good to go:

[root@webserver ~]# perl -V:usemultiplicity -V:usemymalloc
usemultiplicity='define';
usemymalloc='n';

Let's install some prerequisites to embed our Perl:

[root@webserver ~]# yum install perl-CPAN perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker perl-ExtUtils-Embed perl-devel 

Great. Now, let's back up our nginx configuration files and recompile it:

[root@webserver ~]# cp -rf /opt/nginx/conf/ /root/nginx_conf_bak

OK, now we should configure nginx as always, but with the addition of --with-http_perl_module. So in my case it is:

[root@webserver nginx-1.6.2]# ./configure --add-module=../naxsi/naxsi-master/naxsi_src/ --prefix=/opt/nginx --error-log-path=/var/log/nginx/nginx_error.log --http-log-path=/var/log/nginx/nginx_access.log --user=www-data --group=www-data --with-http_addition_module --with-http_geoip_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_realip_module --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_memcached_module --without-http_ssi_module --without-http_uwsgi_module --without-http_scgi_module --with-http_perl_module

If you, like me, like to change the nginx headers and version to spoof their make to something else, say, IIS, then you should also make sure that the #define NGINX_VERSION in /src/core/nginx.h variable consists of numbers and dots only, for example not 3.4.1 (Unix), but 3.4.1, otherwise you'll encounter compilation problems.

Let's proceed to the installation of our new nginx binary that will now support the nginx perl module:

[root@webserver ~]# make
[root@webserver ~]# make install

Let's restore our old config:
[root@webserver ~]# cd /opt/nginx/conf/
[root@webserver ~]# rm -rf *
[root@webserver conf]# cp -rf /root/nginx_conf_bak/* .
[root@webserver conf]# chown -R www-data:www-data *

Time to do some substitutions using regular expressions on our logs. This needs to be in our http block:

[root@webserver ~]# vi nginx.conf
....
http {
perl_set $anonymize_data '
                sub {
                        my $r = shift;
                        my $req =  $r->request_body;
                        $req =~ s/\?card_number\=(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|6(?:011|5[0-9]{2})[0-9]{12}|(?:2131|1800|35\d{3})\d{11})/\?card_number\=XXXX-XXXX-XXXX-XXXX/g;
                        $req =~ s/\&card_number\=(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|6(?:011|5[0-9]{2})[0-9]{12}|(?:2131|1800|35\d{3})\d{11})/\&card_number\=XXXX-XXXX-XXXX-XXXX/g;
                        $req =~ s/\&cvv\=\d\d\d/\?cvv\=XXX/g;
                        $req =~ s/\&cvv\=\d\d\d/\&cvv\=XXX/g;
                        $req =~ s/password\=\w+/password\=XXXXXXXX/g;
                        return $req;
                } ';

Notice that I cover both scenarios; if the card_number arrives first (?card_number); or as a secondary/tertiary variable (&card_number). I do the same for the CVV. Replace "card_number" and "cvv" with whatever variable names your requests come as.

Finally, you will need to replace your "$request" with "$anonymize_data" in your log_format, like so:

log_format main $remote_addr - $remote_user [$time_local] "$anonymize_data" $status $body_bytes_sent "$http_referer" "$http_user_agent";

And now the card number, the CVV and the user's password get masked.

1.2.3.4 - - [02/Dec/2014:14:58:54 +0200] "GET /ccsite/?card_number=XXXX-XXXX-XXXX-XXXX&amount=10000&cvv=XXX&holder_name=N.Ame&expiration_month=1&expiration_year=2015&description=Something&_method=POST HTTP/1.1" 200 395 "https://partnersite.com/apps/ccapp?dothis.js" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0"
1.2.3.4 - - [02/Dec/2014:16:07:46 +0200] "password=XXXXXXXX&username=my_username" 200 416 "-" "Dalvik/1.6.0 (Linux; U; Android 4.3; C1905 Build/15.4.A.1.9)" 

Friday, November 14, 2014

Free Hypervisors Comparison

I often read debates and opinions on the internet concerning the pros and cons of the free versions of the three most widely-spread hypervisors.

Here's my view on them:


ESXi:


Pros:

  • It's the most popular bare-metal hypervisor, so whatever problem you encounter you can just google for the answer.  
  • Supports a huge number of Guest OSes 

Cons:

  • It's not free; this is not even a clever marketing technique. It's a bait and switch, plain and simple. Let me elaborate here. ESXi is free, which is the hypervisor part. Great. But its management tools aren't. So you can't do absolutely anything with it. If you want to actually install a guest OS, you'll need the vSphere client which costs thousands of dollars, or VMware workstation that you will need to use to connect to the ESXi remotely (a solution which will set you back a few hundred dollars instead). 
  • I have found its command line to be obtuse. I know, eye of the beholder and all, but it would be much easier to learn if it had completion for its commands as well as the command parameters, just like XenServer.


Hyper-V:


Pros:

  • If you're planning to install a Windows OS, look no further. Microsoft has done a great job with the optimization and performance of Windows-based VMs. 
 
Cons:

  • Even though it is dead simple to actually install the hypervisor you can't do anything unless you have set up and configured users, domains, firewall rules, etc. from the command line. If you're not already familiar with PowerShell, this can be a daunting and time consuming task (actually, it is a time-consuming task even if you are familiar with the PowerShell). Luckily, there is a script out there that takes care of most of these tasks, reducing the time taken to actually set up everything. After that, you'll need to install the RSAT tools and Hyper-V tools on your PC. Finally, if you're running windows 8, for some strange reason, Microsoft has also decided that you will need the Pro Edition to manage your Hyper-V server.
  • It does not provide zero-downtime live VM migration  
  • Dynamic Memory is only supported with VMs running Windows Vista SP1 and above, or Windows Server 2008 SP2/2008R2 SP1, 2003 R2 SP2 and 2003 SP2.
  • If you're planning to use Linux, you'll need to install the legacy network adapters, and after you're done with the installation of the Linux Integration Services, you'll need to substitute the network adapters for better ones; somewhat minor irritation, I know. 


XenServer:


Pros:

  • It's a fully-fledged Operating System based on Red-Hat 5, which means you can tweak it and do whatever you want. For instance, you will need to have specialized software to take backups of your VMs if you're running Hyper-V or ESXi. You can if you want with XenServer, but personally I've just written a perl script. And it works great. And guess what, too: my backups are written to a GlusterFS volume. If I had ESXi, I would have to use the insecure and deprecated NFS protocol for that. On XenServer, I just had to install GlusterFS, like any other Linux node.
  • Its command line is unbelievably good. And extremely easy to learn. You can do everything with it.
  • The free version is not a watered-down version of the paid one. Almost anything you can do with the XenServer paid version you can with the XenServer free version, except features like GPU virtualization, in-memory read caching, support and patching (you'll have to patch using the command line if you're using the free version).

Cons:

  • XenServer used to have a great networking management interface called the Distributed Virtual Switch Controller. Unfortunately, this tool became deprecated as soon as the product was decided to become open-source. Of course, since XenServer is a fully-fledged Linux distro, this means that it is trivial to actually solve any network-related issue by switching our networking stack to Linux bridge, and taking it from there. 

Sunday, November 2, 2014

Zen Load Balancer Performance Tests Part 2

All right. Time to upgrade the entire thing as described in the "Zen Load Balancer 3.0.3 Perfomance and Security Customization"series.

In summary:

Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 5 was skipped, no NIC bonding was necessary;
Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 4 was implemented, with the 1GB+ recommended settings and no high-memory usage settings;
Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 3 was skipped, no point of checking for invalid packets or rate limit. This is just a test server;
Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 2 was implemented;
Zen Load Balancer 3.0.3 Perfomance and Security Customization Part 1 was implemented (no filesystem tuning though), although the system's memory is 2GB, which means the only benefit here is the NX protection which won't be necessary since this is a test server (really, this should actually produce some overhead).

Let's go over my Zen Load Balancer's stats again:

Memory: 2048MB
CPU #1: 3.0GHz

I decided to use G-WAN as the backend, mainly because:

a) I hadn't used it before and it was a good opportunity to do so;
b) They brag a ridiculously high performance throughput -you know you have to at least check if it's partly true, and if it is, all the better as I'll stress Zen even more;
c) They have an enormous amount of haters; from other webserver fanbois who refuse to accept that a different webserver than the one they use might be better, to company plants that ticket non-existant vulnerabilities to try to prove that G-WAN isn't as good as its publishers claim. This alone can make me install software just out of spite.

So G-WAN it is, then!

The webpage that I will serve will be generated by this little C++ servlet here:
root@gwan:/opt/gwan_linux64-bit/0.0.0.0_8080/#0.0.0.0/csp# cat bench.cpp
// ============================================================================
// C++ servlet example to benchmark Zen and G-WAN Web Application Server   (http://gwan.ch/)
// ----------------------------------------------------------------------------
// bench.cpp: Concatenate a bunch of "Hello world!" until it gets to 400KB (encodings included).
//
// This code is based on the hello.cpp example by Thomas Meitz (gwan at jproxx dot com)
// ============================================================================
// imported functions:
//   get_reply(): get a pointer on the 'reply' dynamic buffer from the server
//   xbuf_cat(): like strcat(), but it works in the specified dynamic buffer
// ----------------------------------------------------------------------------
#include "gwan.h" // G-WAN definitions
#include <string>

using namespace std;

class Hello
{
public:
   void whatsup(xbuf_t* reply, string& str)
   {
      xbuf_cat(reply, (char*)str.c_str());
   }
};
// ----------------------------------------------------------------------------
int main(int argc, char *argv[])
{
   string h("Hello World! ");
   do {
   h+=h;
   } while (h.length() < 240000);
   Hello hello;
   hello.whatsup(get_reply(argv), h);

   return 200; // return an HTTP code (200:'OK')
}
// ============================================================================
// End of Source Code
// ============================================================================


These were the farm settings:



Tests with Zen Load Balancer, customized as per the recommendations in my blog

The test will be the same as before; we'll send 10, 50 and 100 requests concurrently and 10000 in total, all trying to access the output of our script.

The tests will be run 5 times each. After that, we'll find the median value according to that point in time so as to eliminate any statistical aberrations.


D:\apache\Apache24\bin>abs -c10 -n10000 -f TLS1 https://192.168.0.99:443/?bench.cpp
This is ApacheBench, Version 2.3 <$Revision: 1554214 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.0.99 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        G-WAN
Server Hostname:        192.168.0.99
Server Port:            443
SSL/TLS Protocol:       TLSv1,RC4-SHA,1024,128

Document Path:          /?bench.cpp
Document Length:        425984 bytes

Concurrency Level:      10
Time taken for tests:   110.865 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      4262600020 bytes
HTML transferred:       4259840000 bytes
Requests per second:    90.20 [#/sec] (mean)
Time per request:       110.865 [ms] (mean)
Time per request:       11.086 [ms] (mean, across all concurrent requests)
Transfer rate:          37547.54 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4   13   6.9     11     106
Processing:    45   98  24.3     97     756
Waiting:        1   11   8.0     10     103
Total:         57  111  25.0    109     775

Percentage of the requests served within a certain time (ms)
  50%    109
  66%    114
  75%    117
  80%    120
  90%    127
  95%    134
  98%    146
  99%    161
 100%    775 (longest request)

Whoa, got to say, this looks extremely promising. Let's do this four more times, find the median values and create a chart out of it.

This is our Zen's CPU usage for 10 concurrent connections:


This is not bad at all. No excessive CPU usage, and good performance as well for an HTTPS farm; apachebench reported around 90 requests per second.

For 50 concurrent connections:

Insane. Once again, no excessive CPU usage, and good performance for an HTTPS farm; apachebench reported almost 89 requests per second!

For 100 concurrent connections:

And this seals it. With our recommended performance upgrades, Zen is both faster and uses a lot less CPU. Just compare this with our previous results! For the record, apachebench reported around 80 requests per second.