Scripting Queue Depth in a QLogic/EMC environment

First of all, apologies to anyone using Emulex, non-EMC SANs – this guide isn’t for you – I’ll try make a guide at some other stage.

Also, at the end of this guide, I will publish a script that you can run in your Service Console to optimize your SAN performance.

First, lets define some variables that we need to tune in ESX:

Maximum Queue Depth: HBA queue depth is an HBA driver parameter that refers to the number of HBA SCSI command buffers that can be allocated by an HBA port. It relates to the maximum number of outstanding commands that can execute on any one HBA port. When the QLogic driver loads, it registers the value of the queue depth with the SCSI mid-level, which notifies the SCSI layer above the HBA driver how many commands the HBA driver is willing to accept and process. The default for ESX 3.x is 32 and 16 for ESX 2.5.x. The valid queue depth range is from 1–256, but the best value for your environment depends on the following factors:

• The total number of LUNs exposed through the storage array (target) ports.
• The queue depth of the storage array (target) port

The effects of changing the HBA Max Queue Depth parameter are most noticeable when the application workload type is transaction oriented.
Maximum Sectors
Maximum Scatter Gather Entries

The information under Maximum Queue Depth came from here and you can view information on the other two at the same link – they don’t affect performance as much.

Now, the way to determine what the Command Queue Depth/Execution Throttle should be set to, on your device should be calculated by the following formula:

T => P * q * L

where: The target queue depth should be greater than or equal to the product of host paths, HBA queue depth, and the number of LUNs configured.

For heterogeneous hosts connected to the same port on the device, use the following equation:

T => Host OS 1 (P * q * L) + Host OS 2 (P * q * L) + ………… + Host OS n (P * q * L)

P = The number of host paths connected to that array device port.
q = Queue depth per LUN on the hosts (for the host port); i.e., the maximum I/Os outstanding per LUN from the host at any given time. This is a QLogic FC driver/SCSI driver parameter.
L = The number of LUNs configured on the array device port as seen by a particular host path.
T = The maximum queue depth per array device port, which signifies the maximum number of I/Os outstanding (that can be handled) on that port at a given time. The value for T is typically 2048 for EMC arrays.
Q = Queue depth per HBA port; i.e., the maximum I/Os that an HBA can have outstanding at an instant. This parameter is needed only if q (queue depth per LUN) is not available.

So lets do a worked example – lets assume that you are running 6 ESX servers, sharing 8 LUNs on a CX3-20f with dual front-end Fibre ports on each controller. Each fibre card has a zoned connection to each fibre port on both controllers.

Thus:

T=2048

P=1

q is unknown

L=8

So the equation for a single host is:

2048=8q

For 6 hosts, the equation is:

2048=6*8q

Therefore q=42.66

Now normally, you would round this up, but hold off – rounding up can cause a QFULL situation where the array will force your HBA to reduce it’s queue depth to finish processing previous commands.

Now obviously, what we’re seeing here is a situation that would develop if all the ports on an active-active array died and all LUNs migrated to the one port on one controller. This means that what we see is the “safe” queue depth.

If you assume that a port will never die, the effective MRU policy on active/passive arrays states that all LUNs are presented through all ports in one controller, effectively doubling your T and q values (on a CX3-20f) to a queue depth of 85.

However, increasing the queue depth in the QLogic Fibre Channel HBA above 65 has little effect, because the maximum parallel execution (or queue depth) of SCSI operations is 64.

So in this case, all we would do is set the value to 64.

So lets write the script. The requirements to run it are:

• EMC Navisphere CLI installed on the host ESX server’s console
• Assumption of 2 QLogic HBAs per each server attached to the Clariion with two FC switches

Here’s the script:

#!/bin/bash

# Setting global wariables

numAllLUNs=`/opt/Navisphere/bin/navicli -h $1 getlun | grep Name | grep -v MirrorView | wc -l | sed 's/\ //g'`
numHotSpareLUNs=`/opt/Navisphere/bin/navicli -h $1 getlun | grep "Hot Spare Ready" | wc -l | sed 's/\ //g'`
numHBAConnections=`/opt/Navisphere/bin/navicli -h $1 getall | grep "Total number of initiators" | sed 's/Total\ number\ of\ initiators\:\ //g;s/\ //g'`
numSPPorts=`/opt/Navisphere/bin/navicli -h $1 getall | grep "SP UID:" | grep -v iqn | wc -l | sed 's/\ //g'`

# Checking Navisphere is installed

rpm -qa > /var/rpmdump
if ! grep "navi" /var/rpmdump >> /dev/null ; then
        rm -f /var/rpmdump
        echo ""
        echo NaviCLI is not installed - please install NaviCLI
        exit
fi
rm -f /var/rpmdump

# Rescan all HBAs

for HBA in $(esxcfg-info -w | grep vmhba | awk '{print $3}' | grep -e 'vmhba\+[1-9]' -o)
do
        esxcfg-rescan ${HBA} >> /dev/null
done

# Doing FC calculations

if grep "QLogic" /etc/vmware/esx.conf >> /dev/null ; then
        numHBAsESX=`grep "QLogic" /etc/vmware/esx.conf | wc -l | sed 's/\ //g'`
        if [ $numHBAsESX != 2 ] ; then
                echo ""
                echo There is more/less than 2 QLogic HBAs in this server... exiting
                exit
        fi
        echo""
        echo "Checking: You have QLogic HBAs - good."
        regHBAs=`echo $numHBAConnections 2 | awk '{ print $1 / $2 }'`
        echo ""
        echo The number of registered HBAs on your EMC array is $regHBAs
        numFEPorts=`echo $numSPPorts 2 | awk '{ print $1 / $2 }'`
        echo ""
        echo Your array has $numSPPorts FE fibre ports - that is $numFEPorts per controller
        TQueueDepth=`echo $numFEPorts 2048 | awk '{ print $1 * $2 }'`
        echo ""
        echo This means your effective target queue depth is $TQueueDepth
        numLUNs=`echo $numAllLUNs $numHotSpareLUNs | awk '{ print $1 - $2 }'`
        echo ""
        echo You have $numLUNs total LUNs, not including hot spares
        MaxQueueDepth=`echo $TQueueDepth $numLUNs | awk '{ print $1 / $2 }' | sed 's/\..*//g'`
        echo ""
        echo This means your command queue depth is $MaxQueueDepth - rounded down to nearest whole integer
        if [ $MaxQueueDepth -ge 64 ]; then
                echo ""
                echo Because the Queue Depth is greater than 64 the Queue Depth will be set artifially
                echo set to 64 in accordance with QLogic best practices
                /usr/sbin/esxcfg-module -s ql2xmaxqdepth=64 qla2300_707_vmw
                echo ...Done
        else
                echo ""
                echo Setting Queue Depth to $MaxQueueDepth
                /usr/sbin/esxcfg-module -s ql2xmaxqdepth=$MaxQueueDepth qla2300_707_vmw
                echo ...Done
        fi
        /usr/sbin/esxcfg-advcfg -s 128 /Disk/SchedNumReqOutstanding >> /dev/null
        /usr/sbin/esxcfg-advcfg -s 1 /Disk/UseLunReset >> /dev/null
        /usr/sbin/esxcfg-advcfg -s 0 /Disk/UseDeviceReset >> /dev/null
        echo ""
        echo Reconfiguring your intird
        /usr/sbin/esxcfg-boot -b
        echo ...Done
else
        echo "You don't have QLogic HBAs. Sorry."
fi

The script is available for download here.

To run it, first chmod a+x it, then run:

sh EMCQueueDepth_set.sh SPAorSPBIPAddressofEMCArray

This script will query the array for all required information while assuming that there are two switches in the environment, to provide even numbers of initiators on the array.

This script will cause problems if there are no switches in your environment or if you have odd numbers of single-path connections to the SAN

Caveat:

This has only been tested twice. Do not come running to me if this causes issues – I simply made the script to run as a cron job in my test lab. If you want to use it – you do so at your own risk.

Nonetheless, if you have a successful test, please let me know and suggest ideas for improvement.

Also, I completed this script from 2am-3am in the morning so if my math doesn’t make sense or my Queue Depth calculations sound like utter BS, let me know.

Cheers,

Leo