Home‎ > ‎

Queue Depth VMware SCSI Controllers

There has always been a lot of questions thrown at me! What is the best controller? SATA? SCSI? IDE ?? The answer is SCSI obviously. But remember that there are different SCSI Adapter Types. Now, lets analyze what has better queue depth.

With ESXi 6.0/ESXi 6.5 there are four different options of SCSI Controllers. Refer the Following table -

Adapter Type

OS Type

Minimum Requirements

Maximum SCSI Adapters

Use Cases

BusLogic Parallel

Server

-

4

15 devices per controller, issues with 64-Bit OS, 2TB VMDK limit, VMware suggests to migrate off this adapter

LSI Logic Parallel (formerly LSI logic)

Server/Desktop

-

4

15 devices per controller, required for Microsoft Clustering Service (older than Windows 2008)

LSI Logic SAS

Server/Desktop

HWv7

4

15 devices per controller, with most of OS the standard SCSI controller, required for MSCS (Windows 2008 or newer)

PVSCSI

Server

HWv7

4

15 devices per controller, lower CPU cost in many higher I/O use cases, suggested for high I/O use cases, no MSCS support for ESXi 5.5 U2 and lower



To maximize performance virtual disks should be distributed across multiple vSCSI adapters. A maximum of 4 vSCSI adapter can be configured per VM with a maximum of 15 vDisks per vSCSI adapter. By using multiple vSCSI adapters you open up more I/O queues. The following table shows the queue depths when using the PVSCSI adapters as compared to the LSI Logic SAS adapter.

Queue
PVSCSI
LSI Logic SAS
Default Adapter Queue Depth
256
128
Maximum Adapter Queue Depth
1024
128
Default Virtual Disk Queue Depth
64
32
Maximum Virtual Disk Queue Depth
254
32

Tuning of PVSCSI

There is a very good KB: VMware KB 2053145 you should follow to understand how to tune the different values in ESXi itself as well as the VM. Two settings can be tuned:


  • Adjust the queue depth for the HBAs on the ESXI host
  • Increase the PVSCSI queue inside the Windows or Linux guest


Note: The default number of ring pages is 8, each of which is 4 KB in size. One entry of 128 bytes is needed for a queued I/O, which means that 32 entries on a single page of 4096 bytes, resulting in 256 entries (8x32). The Windows PVSCSI driver adapter queue was hard coded in previous releases,  but it can be adjusted up to the maximum of 32 pages since the versions delivered with VMware Tools.

This note I got out of the KB itself but what does that mean? Let me explain what Ring Pages and Queue Depth really stands for as it often gets understood wrong.


Ring Pages:
The 128 byte entry describes the I/O. It is not the actual memory to which the I/O is being directed to/from. In other words, the pages that are used for ring buffer are used to describe the actual I/O operation. They are not used for the actual I/O itself. Think of them as having pointers to the pages that will be used for DMA operation. A portion of a page (non-ring pages) may be used for one I/O and another portion (possibly even overlapping) may be used for another I/O. So the latter is possible.


Queue Depth:
The queue depth is a number that in the case of PVSCSI reflects the limits of the adapter.
The adapter uses 8 ring pages and thus can support a queue depth of 256. It is really an artificial number since PVSCSI is not a real device because it is a VMware paravirtualized SCSI device. However for other devices (real adapters), it reflects an actual HW limit. The ring is in the hardware and it has a limit and hence the queue depth! The queue depth is per adapter. So if you have 4 PVSCSI or any LSI adapter for that matter you will get 4 * Queue Depth. As a consequence, there will be 4 * Ring Pages as well.


LSI Logic:
It is a similar construct. The only difference is that the hardware controller has an upper limit based on real queue resources on the HBA. However, the driver may artificially allow you to use a lower value than what the hardware can support. If the driver lets you set it to a higher value than what the HBA claims then it will be queued in the driver instead of being queued in the ESXi SCSI mid-layer queues. It will be accounted as part of the queueing delays in that case.

Comments