vSphere 5 hardware dependant iSCSI vs software iSCSI on Broadcom NetXtreme II 5709c

Recently I've been setting up vSphere 5 infrastructure with 2 iSCSI storages.
My first impression was to set up ESXi with hardware dependant iSCSI to offload CPU and make performance a bit better. So I set up everything, connected strorage to hosts, checked it worked and then tried to switch to 9000 bytes MTU. First of all I set up hardware switches to iSCSI mode (jumbo frames, storm protection off, etc.). Storage was allready set to support jumbo frames. Then I set up vSwitches, NICs and port groups ... and I lost connection to my storage. That was strange and esxi logs was saying something about frame size mismatch. To be honest there was not too much about it over the internet. But I found a clue that Broadcom 5709 doesn't support jumbo frames in hardware dependant mode. That is strange.

So I switched all hosts to software iSCSI and then set up jumbo frames again. And this time everything worked fine after switching to 9000 MTU, so that is true that hw dependant mode with this NIC does not support jumbo frames. Time to check performance to get the idea which mode is better (I was guessing big MTU one).

Here are most important bonnie++ results (I've checked more raid modes and VM setups, but they are irrelevant).

MTU 1500 hardware dependant iSCSI:
Mpath round robin (vmware) - storage R50 10 SATA drives:
15744M,,,,41656,3,38281,3,,,132610,5,1346,15

Mpath round robin (vmware) - R10 10 SATA drives:
15744M,,,,57688,4,33483,3,,,84875,3,818.5,21

MTU 9000 software iSCSI:
Mpath round robin (vmware) - storage R50 10 SATA drives:
15744M,,,,151118,11,72237,6,,,152371,5,1387,14

Mpath round robin (vmware) - R10 10 SATA drives:
15744M,,,,157955,13,58615,4,,,149033,5,871.5,21

So it's clear that jumbo frames are much better even if using software iSCSI. Write and read performance in this mode is limited by 1Gb NICs speed (with round robin is a bit faster than 1Gb) and it cannot go faster even this storage is much much faster in sequential operations. Without jumbo frames in hw dependant mode performance was poor. More than 3 times slower on writes, almost 2 times slower in rewrites, bit slower R50 reads and much slower R10 reads. This shows that you should deffinitely go with jumbo frames instead of hw dependant iSCSI if your hardware is limited to 1500 MTU in this mode.

Another conclusion is that iSCSI storage is almost always limited by NICs speed. So if you can afford FC storage go for it.

Comments

I couldn't resist commenting. Very well written!

Hey there! Do you use Twitter? I'd like to follow you if that would be okay. I'm absolutely enjoying your blog and look forward to new posts.

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Main menu

Article | by Dr. Radut