Enabling HBA mode on my dl380 g6 with RHEL 8

If you are familiar with the HP server line, you likely know that the DL380 G6 (as well as many other models of their servers), you would know that the integrated RAID controller, the P410i, doesn’t support HBA mode for disks. Since I needed more storage then anything, and don’t want to spend a fortune on high capacity SAS disks, I decided to use SATA.

A common issue with this particular controller is that it has a battery backup that seriously hurts performance when it is dead. It also is AWFUL for any modern storage system, that relies on commodity disks in place of hw raid on a proprietary controller. It’s a pain and very restrictive.

The solution used to be to get a different RAID controller, and just avoid the integrated one. Unfortunately, the system will only boot off of 3 different locations:

  • The integrated RAID controller
  • The ONLY SATA port on the mainboard, specifically used for the optical drive
  • A USB drive or memory stick

This made me come up with a different solution. I took out the RAID expander (allows the p410i to address up to 16 disks), and replace it with a patched LSI controller (LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]). This allowed me to address 8 of the disks separately. I then installed a 16gb patriot usb drive (USB2.0) into the internal USB port, to act as the main boot drive. This is where the bootloader, kernel and initramfs live, and let’s the system continue to boot.
I have 2 SSDs installed on that card, which is holding the following filesystems:

  • / – RAID1 (mirrored) for redundancy
  • /var – RAID1 (mirrored) for redundancy
  • /home – RAID0 for speed. This is only for local users, and will not be used much at all
  • swap – RAID0 for speed. Not needed much, since I have 128GB of ECC memory installed.

It wasn’t that easy of course. I had to add a “Driver update disc” to get the card recognised properly, as RHEL 8 disabled the device in the mpt3sas driver that’s included. This meant adding the following to the kernel parameters on the RHEL installer boot:

inst.dd=https://elrepo.org/linux/dud/el8/x86_64/dd-mpt3sas-34.100.00.00-1.el8_3.elrepo.iso

This allowed the installer to recognise the device, and installed the updated driver permanently, so it would continue to work after the installation was done.

It’s booting stably, and made me ready for the next part: Enabling HBA mode on the internal controller.

I used a medium post by Terryjx to get started, and figure out what I needed. After reading through the entire article, I decided I would use a 5.4 kernel on this system, so I can use the patched driver. That meant using the elrepo kernel.

To enable that kernel, run the following commands:

$ sudo dnf -y install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
$ sudo dnf --allowerasing --enablerepo="elrepo-kernel" install kernel-lt{,-devel,-headers,-modules,-modules-extra}

As of this writing, the current lts kernel in elrepo is 5.4.98-1.el8.elrepo.x86_64. I then rebooted to make sure that kernel worked, which it did without issue. Didn’t even need to install a new mpt3sas driver, as the correct on is included in the modules.

I was now ready to enable hba mode, but that required that I have the 6.64 firmware for the p410i controller. I used the link from the above article: https://downloads.hpe.com/pub/softlib2/software1/sc-linux-fw-array/p332076214/v110820/hp-firmware-smartarray-14ef73e580-6.64-2.i386.rpm. I just had to install it via:

$ sudo dnf install hp-firmware-smartarray-14ef73e580-6.64-2.i386.rpm
$ cd /usr/lib/i386-linux-gnu/hp-firmware-smartarray-14ef73e580-6.64-1.1/
$ sudo cp ccissflash /usr/bin/ccissflash
$ sudo bash hpsetup

It will take a while to complete. Just answer it’s questions and let the update complete. You’ll have to reboot once it’s done to allow the firmware to recognise the change.

To do the next few steps, I needed a couple of other repositories. Namely, the EPEL repo, and the RHEL equivalent to the CentOS PowerTools repo, called CodeReady. This took a bit to figure out, as whenever you google for RHEL PowerTools equivalent, it just comes up with PowerTools. I eventually had to search each repository to find the correct one with dkms and pandoc. Yes, dkms is required to ensure the custom module is built and installed with each kernel update.

$ sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
$ sudo dnf config-manager --set-enable codeready-builder-for-rhel-8-x86_64-rpms
$ sudo dnf group install "Development Tools"
$ sudo dnf install dkms pandoc

In order to install the new driver, you need to be root to enable dkms building. A symlink will be created from the cloned repository to the dkms directory in the source tree, allowing you to build the module.

$ sudo -i
# git clone -b dkms https://github.com/artizirk/hpsahba
# cd hpsahba
# make ##Build the helper program hpsahba
# ./hpsahba -E /dev/sg0 ##Use the correct location of your controller, /dev/sgN where N is the number
# cd contrib/dkms
# ./patch.sh 5.4
# dkms add ./
# dkms install --force hpsa-dkms/1.0
# modprobe -r hpsa ##Remove the existing module
# modprobe -v hpsa hpsa_use_nvram_hba_flag=1
# echo 'options hpsa hpsa_use_nvram_hba_flag=1' > /etc/modprobe.d/hpsa.conf
# cp -v /boot/initramfs-$(uname -r).img /root ##Backup original initramfs
# dracut -f -v ##Generate a new initramfs

Let’s explain what happened there a bit, line by line

  1. Become root
  2. Clone the repository containing the patches for the hpsa driver, and the hpsahba program source
  3. Enter the directory that was cloned
  4. Build the hpsahba software
  5. Enable HBA mode on your p410i (make sure the firmware is version 6.64)
  6. Enter the dkms directory to setup the module build
  7. Run the provided script to patch the given kernel version. Check the repo’s kernel directory for supported versions.
  8. Add the patched module to dkms
  9. Build and install the patched module. –force is required in order to replace the in tree module with the patched one.
  10. Unload the in tree hpsa module
  11. Load the patched hpsa module. The hpsa_use_nvram_hba_flag=1 is required for this to work
  12. Added the hpsa module load options to make sure it always loads with that flag
  13. Backup the original initramfs
  14. Generate a new initramfs with the patched module.

You should now be able to to do an lsblk and see all of the attached disks. For example, my lsblk on the server I did this on:

NAME            MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda               8:0    0   1.8T  0 disk
sdb               8:16   0   1.8T  0 disk
sdc               8:32   0 931.5G  0 disk
sdd               8:48   0 931.5G  0 disk
sde               8:64   0 931.5G  0 disk
sdf               8:80   0 931.5G  0 disk
sdg               8:96   0 931.5G  0 disk
sdh               8:112  0 931.5G  0 disk
sdi               8:128  0   1.8T  0 disk
sdj               8:144  0 931.5G  0 disk
sdk               8:160  0 931.5G  0 disk
sdl               8:176  0   1.8T  0 disk
sdm               8:192  0   1.8T  0 disk
sdn               8:208  0   1.8T  0 disk
sdo               8:224  0 232.9G  0 disk
└─sdo1            8:225  0 137.1G  0 part
  └─md127         9:127  0   274G  0 raid0
    ├─rhel-root 253:0    0    70G  0 lvm   /
    ├─rhel-swap 253:1    0     4G  0 lvm   [SWAP]
    ├─rhel-var  253:2    0   100G  0 lvm   /var
    └─rhel-home 253:3    0   100G  0 lvm   /home
sdp               8:240  0 232.9G  0 disk
└─sdp1            8:241  0 137.1G  0 part
  └─md127         9:127  0   274G  0 raid0
    ├─rhel-root 253:0    0    70G  0 lvm   /
    ├─rhel-swap 253:1    0     4G  0 lvm   [SWAP]
    ├─rhel-var  253:2    0   100G  0 lvm   /var
    └─rhel-home 253:3    0   100G  0 lvm   /home
sdq              65:0    1  14.9G  0 disk
└─sdq1           65:1    1  14.9G  0 part  /boot

That’s 8 drives attached to the LSI controller, and 8 attached to the integrated p410i, exposed directly to the OS.

I have no benchmarks for this yet, but can confirm it’s stable and working exactly how I expect it to.

Published by Matthew Stobbs

Matthew is the Chief Information Officer and Chief Financial Officer for Sprouting Communications. From server side development, to Web Applications, to Phone Apps, his passion for computing ensures he is on top of the latest technologies and design trends.

Leave a comment

Your email address will not be published. Required fields are marked *