Dell PowerEdge, pfSense and RAID status monitoring using ports/pkg

The Dell PowerEdge is a thing of beauty. Not only can you find a second-hand one for next to nothing, but they are great for acting as load balancers, firewalls and routers. With the the right tools and enough expertise you can build systems that the big players sell for tens, or sometimes hundreds of thousands of dollars. All you need to have is a second-hand Dell PowerEdge, a few Intel NICs, the right open-source software and you're set.

So I have this Dell PowerEdge box that acts as a firewall, having pfSense 2.1 installed on it. How do we go about monitoring its RAID status? The answer is pretty straightforward; Install megacli on a virtual machine, create a package out of it, transfer it to our pfSense box and install.

a) Check your FreeBSD version on your webconfigurator dashboard and depending on whether you have the 32 or the 64-bit version of pfSense installed, download the right .iso from  http://ftp.freebsd.org/pub/FreeBSD/. For instance, I have pfSense 2.1, which is based on FreeBSD 8.1 and is 64-bit. Therefore, I downloaded this:  http://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/8.1/FreeBSD-8.1-RELEASE-amd64-livefs.iso.
b) Create a virtual machine on your local PC using your favourite virtualization solution. Make sure it's 32-bit if your pfSense is 32-bit or 64-bit if your pfSense is 64-bit.
c) Install using the downloaded .iso.
d) Install all available packages, we might need to use this method again in the future for other packages.
e) If FreeBSD installation throws an error about not being able to find your distribution in the available mirrors, that would be because it's an older one. Just choose ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/ if you have a 32-bit system or ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/amd64/ if you have a 64-bit system as the URL of the FreeBSD distibution on the remote ftp site.
f) Time for the actual work:

freebsd# freebsd-update fetch
freebsd# freebsd-update install
freebsd# portsnap fetch
freebsd# portsnap extract
freebsd# portsnap fetch update
freebsd# cd /usr/ports/sysutils/megacli/
freebsd# make package-recursive
Creating bzip'd tar ball in '/usr/ports/sysutils/megacli/work/megacli-8.07.07.tbz'

g) Go to your pfSense Webconfigurator, System, Advanced, Admin Access and make sure "Enable Secure Shell" is checked.
h) Upload the created bzip'd tar ball to your pfSense using SFTP or SCP.
i) SSH to your pfSense box and choose 8 (shell).
j) I have uploaded the packed to the /root directory so:

[2.1-RELEASE][root@pfsense.mynetwork.net]/root(1): pkg_add /root/megacli-8.07.07.tbz
[2.1-RELEASE][root@pfsense.mynetwork.net]/root(2): pkg_info
bsdinstaller-2.0.2013.0911 BSD Installer mega-package
gettext-0.18.3      GNU gettext package
libiconv-1.14_1     A character set conversion library
megacli-8.07.07     SAS MegaRAID FreeBSD MegaCLI

k) Now, you can go to your pfSense webconfigurator: System, Packages, Available Packages and install "mailreport".
l) Then go to System, Advanced, Notifications and configure your mail server SMTP settings.
m) Finally, go to Status, Email Reports and create a new mail report.
n) Configure it, save it and then edit it so that it executes the /usr/local/sbin/MegaCli -LDInfo -Lall -aALL and the /usr/local/sbin/MegaCli -PDList -aALL commands.

Done! Now every day you'll get a status report for your firewall's disks!

Note 1: This method uses the deprecated ports/pkg method. I will describe the method required using the newer pkgng system in a future post.
Note 2: Megacli does not work with the older SAS 6/iR cards, you will need the mptutil package. The method is the same.

Monitor your HP Smart Array RAID controller from XenServer 6.2

I have a few HP Proliant Servers that I use as XenServer hosts. The problem is that I want to know the health of my disks and my RAID arrays, and of course I can't get those from the XenServer guests.

So here's what we do:

Just google for knowledge base search hp support center (HP changes their links too often for me to give you the current link, it will probably change in a few months), follow that link and then choose "Search HP Support Center" in Knowledge Base.

When you're there search for HP Array Configuration Utility CLI for Linux and select the most recent 32-bit version (XenServer Dom0 is 32-bit).

Get its URL and download it from your XenServer or download it to your PC and sftp it to your Xenserver. The most recent version at the time of writing was this:

http://ftp.hp.com/pub/softlib2/software1/pubsw-linux/p414707558/v71530/hpacucli-9.10-22.0.i386.rpm

Time for the fun part. Let's log into our XenServer:

[root@server ~]# wget http://ftp.hp.com/pub/softlib2/software1/pubsw-linux/p414707558/v71530/hpacucli-9.10-22.0.i386.rpm

From the README.txt:

Description
-----------

  The Array Configuration Utility CLI is a commandline-based disk
   configuration program for Smart Array Controllers and
   RAID Array Controllers.
 
* All other product names mentioned herein may be trademarks of their
  respective companies.

Supported Controllers

  Smart Array products:
     Smart Array 5312 Controller
     Smart Array 5302 Controller
     Smart Array 5304 Controller
     Smart Array 532 Controller
     Smart Array 5i Controller 
     Smart Array 641 Controller
     Smart Array 642 Controller
     Smart Array 6400 Controller
     Smart Array 6400 EM Controller
     Smart Array 6i Controller
     Smart Array P600 Controller
     Smart Array P400 Controller
     Smart Array P400i Controller
     Smart Array E200 Controller
     Smart Array E200i Controller
     Smart Array P800 Controller
     Smart Array E500 Controller
     Smart Array P700m Controller
     Smart Array P410i Controller
     Smart Array P411 Controller
     Smart Array P212 Controller
     Smart Array P712m Controller
     Smart Array B110i SATA RAID
     Smart Array P812 Controller
     Smart Array P220i Controller
     Smart Array P222 Controller
     Smart Array P420 Controller
     Smart Array P420i Controller
     Smart Array P421 Controller
     Smart Array P822 Controller
     Dynamic Smart Array B320i RAID
     Dynamic Smart Array B120i RAID

  MSA products:
     MSA500 Controller
     MSA500 G2 Controller
     MSA1000 Controller    
     MSA1500 CS Controller
     MSA20 Controller  

[root@server ~]# yum install -y --nogpgcheck hpacucli-9.10-22.0.i386.rpm
[root@server ~]# hpacucli controller slot=1 physicaldrive all show
Smart Array P410 in Slot 1

   array A

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 1 TB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 1 TB, OK)

   array B

      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA, 1 TB, OK)
      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA, 1 TB, OK)

   array C

      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 1 TB, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 1 TB, OK)

   array D

      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 1 TB, OK)
      physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 1 TB, OK)

   array E

      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SATA, 1 TB, OK)
      physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SATA, 1 TB, OK)

And a few helpful commands:

[root@server ~]# hpacucli help
CLI Syntax
   A typical ACU CLI command line consists of three parts: a target device, 
   a command, and a parameter with values if necessary. Using angle brackets to
   denote a required variable and plain brackets to denote an optional 
   variable, the structure of a typical ACU CLI command line is as follows:

      <target> <command> [parameter=value]

   <target> is of format:
      [controller all|slot=#|wwn=#|chassisname="AAA"|
                  serialnumber=#|chassisserialnumber=#|ctrlpath=#:# ]
      [array all|<id>]
      [physicaldrive all|allunassigned|[#:]#:#|[#:]#:#-[#:]#:#]
      [ssdphysicaldrive all|allunassigned|[#:]#:#|[#:]#:#-[#:]#:#]
      [logicaldrive all|#]
      [enclosure all|#:#|serialnumber=#|chassisname=#]
      [licensekey all|<key>]
      [ssdinfo]
      Note 1: The #:#:# syntax is only needed for systems that
              specify port:box:bay. Other physical drive targeting
              schemes are box:bay and port:id.
      Note 2: The chassisserialnumber is known in ACU as the
              RAID Array Serial Number. The chassisname is known
              in ACU as the RAID Array ID.
      Note 3: ctrlpath=#:# maps to "smart enclosure hooked up to 
              host bus adapter slot:host bus adapter port"

   Example targets:
      controller slot=5
      controller chassisname="Lab C"
      controller serialnumber=P21DA2322S
      controller chassisserialnumber=9J3CJN71XDCH
      controller wwn=500308B300701011
      controller slot=7 array A
      controller slot=5 logicaldrive 5
      controller slot=5 physicaldrive 1:5
      controller slot=5 physicaldrive 1E:2:3
      controller slot=5 ssdphysicaldrive all
      controller slot=5 enclosure 4E:1 show
      controller slot=5 licensekey XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

For detailed command information type any of the following: 
   help add
   help create
   help delete
   help diag
   help modify
   help remove
   help shorthand
   help show
   help target
   help rescan
   help version

Help also accepts commonly used CLI parameters and ACU keywords. Adding
additional keywords will further filter the help output. Examples: 
   help ssp        (shows all ssp help including show and modify commands)
   help ssp modify (restricts ssp help to only modify commands)
   help migrate
   help expand   
   help extend   
   help <keyword> <keyword> ... <keyword>

   Please note that beginning with ACU/ACUCLI version 8.55, the term 
   "stripe size" has been replaced by "strip size."  This is a change
   of labeling and does not signify a change in functionality.  When 
   distributing data across multiple physical drives (striping) the
   "strip size" is the amount of data that is written to each physical
   drive.  The "full stripe size" refers to the combined size of all
   the strips across all physical drives,  excluding parity-only drives.

Done!