VMware ESX iSCSI/Cisco/EMC AX4-5i jumbo frames

Update: It appears that I myself did not read the manual – as pointed out by stormlight in the comments, making me the monkey

I’ve changed the title of the post and edited it a bit, because quite clearly, the people who did the implementation were not monkeys for not enabling iSCSI jumbo frames (though they did stuff up a few other things, the partitioning being the most obvious)

This is unsupported, but it does work, so if you want to take the risk and gain a speed increase in your iSCSI traffic, go for it.

—

I was asked to cast an eye over a friend’s installation (works as a sys admin – Dell Australia Professional Services did the implementation a fair while ago). He has two ESX 3.5 Update 1 servers attached to an iSCSI Dell|EMC Clariion AX4-5i.

Very simple config. But no jumbo frames configured.

Here’s what I did:

1. Upgraded him to 3.5 Update 2
2. Configured the two Cisco switches with jumbo frame support
3. Configured his vSwitch with jumbo frame support
4. Configured his VMkernel with jumbo frame support
5. Configured the AX4-5i with jumbo frame support

Obviously, I’m not going to explain how I did 1. But I will explain how to do 2-5

Configuring Cisco Layer-3 switches with jumbo frame support

• Log into your switch and go into conf t mode.
• Run: system mtu jumbo 9000

At this point your switch will ask to reboot. Obviously, this is something you will need to do.

Configure vSwitch with jumbo frame support

Select the vSwitch for your software iSCSI VMkernel adapter – for the purpose of this, let us assume it is vSwitch2

• Run: /usr/sbin/esxcfg-vswitch -m 9000 vSwitch2

Configure VMkernel iSCSI software adapter with jumbo frame support

In this case we are assuming that the VMkernel adapter in question will reside on vSwitch2 (as above) and will be called iSCSI

• Run: /usr/sbin/esxcfg-vswitch -A iSCSI vSwitch2 to add the iSCSI portgroup to vSwitch2
• Run: /usr/bin/esxcfg-vmknic -a -i 192.168.0.1 -n 255.255.255.0 -m 9000 iSCSI

Obviously, in the above, change your IP address and netmask to the right one (bear in mind the iSCSI software adapter must reside on the same network as the iSCSI target)

Configure the AX4-5i with jumbo frames support

Bear in mind that these instructions work for the CX3- series of arrays but not on the AX100, AX150i, the CX300i or the CX500i. Furthermore at least FLARE R24 is required for this, as well as Navisphere SecureCLI if the array has Navisphere Express. The following are the Navisphere Express instructions:

• Log into a server with Navisphere SecureCLI installed
• Run: naviseccli -h <SPx_IP_address> -user <username> -password <username> -scope 0 connection -setport -sp a -portid 1 -address <xxx.xxx.xxx.xxx> -subnetmask <xxx.xxx.xxx.xxx> -gateway <xxx.xxx.xxx.xxx> -mtu <nnnn>

Obviously in the above case, you’re logging onto the SP IP identified by <SPx_IP_address> with username <username> and with password <username>. What you then do is set the SPA port ID of 1 with an IP, subnet mask and gateway and an MTU.

You have to do this for all all ports on all SPs by varying the above information in the command. Remember that on each SP (A and B) there are two iSCSI ports (0 and 1). Obviously in this case we must set the MTU to 9000 but other values are available (1500, 1548, 2000, 2450, 3000, 4000, 4080, 4470, 5000, 6000, 7000, 8000, 9000)

If you have Navisphere Manager on your array, the process is much simpler:

1. Right-click the array icon.
2. Select “iSCSI” and then “Port Management.
3. In the Port Management screen, select a path and click “Properties.”
4. In the Properties windows, click the drop down button for MTU and select the highest transmission rate that is supported on the SAN.
5. Click Apply.

—

If you want to, you can also check that the jumbo frames can traverse your network with the MTU size of 9000: vmkping 192.168.0.20 -s 9000

In the above command, the assumption is that the iSCSI target has the IP of 192.168.0.20. Change as required

Caveats: As mentioned earlier, you will need to reboot your Cisco switch for jumbo frames support. Also, changing your MTU settings on the EMC array on the fly will cause the iSCSI NICs to reboot – all devices linked to them will be logged out. BE CAREFUL!

All done