Problem Statement
Use of HP FlexFabric 10Gb 2-port 534FLB Adapter in a vSphere design using dependent hardware iSCSI mode.
Need to find out the number of outstanding iSCSI I/Os that the adapter can handle, to ensure that number of concurrent SCSI commands has been taken into account.
Methodology
HP FlexFabric 10Gb 2-port 534FLB is based on Broadcom 57810S chipset which uses the bnx2i driver/firmware (see March 2014 VMware FW and Software Recipe)
Broadcom’s bnx2i iSCSI driver is dependent hardware iSCSI (see Cormac Hogan excellent blog post on vSphere 5.1 Storage Enhancements – Part 5: Storage Protocols)
A dependent hardware iSCSI adapter is a third-party adapter that presents itself as a normal NIC, but has an iSCSI offload engine. It requires the use of VMKernel interface, which is then tied to the vmhba (HBA).
Solution
The following applies to the Broadcom 57810S chipset:
- Total outstanding iSCSI Tasks (I/O) per port = 4096 (4K)
- Total iSCSI Sessions per port = 128 – 2048 depending on the Operating System (Host limited)
Each iSCSI Session facilitates communication with a different Target:
- Total of 512 outstanding iSCSI Tasks (I/Os) per Session
Therefore using HP FlexFabric 10Gb 2-port 534FLB Adapter we can have 1024 outstanding iSCSI Tasks across two adapters of 512 each.
VMFocus 
Do you find that the 534FLB Adapter can handle more outstanding requests then the NC553m 10Gig adapters? I’m using the NC553m with the HP customized image in ESXi v5.5, and the adapter (mezzanine) has been a pain in the rear to configure iSCSI CHAP on. HP’s LOM card will configure fine, but the Mezz card isn’t customizable in the Web console.
Also based on your solution I don’t know what you are trying to say? What should be reconfigured to take the 1024 outstanding tasks into consideration?
Hi Charlie, thanks for reading the blog post. The 534FLB Adapter is Broadcom based and the NC553m is Emulex based, you would need to contact HP/Emulex to find out the details.
The 1,024 refers to the number of active SCSI commands a single two port 534 can handle on ESXi Host before queueing occurs which will lead to performance degradation. With each adapter being able to ‘cope’ with 512 active SCSI commands.
By default the ESXi SATP will send 1,000 IOPS down path A to adapter A and then the next 1,000 IOPS down path B to adapter B. This isn’t an issue if your VM aren’t very active as it may take a number of seconds/minutes before path switch over occurs.
If you have an ESXi Host with VMs with very active disks, then you could be in a situation where they are asking for 1,000 IOPS concurrently which would mean queueing of 488 IOPS on 534 adapter A.
Hope that makes sense
Craig – thanks for the clarification. That does make sense. We have a lot of VM’s (Red Hat / Oracle Clusters) that hammer our datastores and we frequently have a lot of IO contention. To help relieve bottlenecks we provided multiple iSCSI vmhba nics, enable jumbo frames, and enable round robin across the hba’s. Also we provided smaller luns (1TB) instead of >2TB luns, so that there would be more IQN’s available to the hosts. So that larger luns wouldn’t be essentially asked to house more .vmdk drives.
I’d be interested in what ESXTOP command you are using to monitor IOP’s and if Queueing is occurring. Also any article that shows what is bad considered “bad Queueing”. My ESXi monitoring indication usually is that 20-50ms latency is bad, and >50ms is considered “unresponsive.”
No worries Charlie, glad it made sense. In terms of monitoring I look at the following in ESXTOP:
1. Storage Adapter Queue Depth – Press D and look at the AQLEN Field e.g. 1024 this is the number of active SCSI commands it can handle
2. Look at the DAVG/cmd line this is the Storage Device latency (anything from HBA onwards to Storage Controller). Anything 25ms + is bad
3. DQLEN is found under – U and the is the number of active SCSI commands each LUN can handle. Note if you have more than one path then this number needs to be multiplied by the number of paths.
Use the information to try and ascertain where the issue is. My advice is to do the maths. For example if you have two HBA with an Adapter Queue Depth of 1024 each, 2048 total.
An ESXi Host has 10 datastores with a DQLEN of 128 each then you are within thresholds as you have 1280. If you are getting performance issues, it could be that you need to increase the number of datastores from 10 to 14 and move some VM’s around.
If the above is still looking OK. Think about your SAN target ports if each controller has 2 ports with a queue depth of 1024 each can it cope with all of the HBA queue depths? For example:
4096 total queue depth on SAN target port but you have 10 x ESXi Hosts each with 1024 Active SCSI Commands then the SAN is the problem.
Hope that helps.