10G Ethernet: More Than a Big Pipe
by Johan De Gelas on November 24, 2010 2:34 PM EST- Posted in
- IT Computing
- Networking
- 10G Ethernet
Exar's Neterion Solution
SR-IOV will be supported in ESX 5.0 and the successor of Windows Server 2008. Since VMware’s ESX is the dominant hypervisor in most datacenters, that means that a large part of the already virtualized servers will have to wait a year or more before they can get the benefits of SR-IOV.
Exar, the pioneer of multiple devices queues, saw a window of opportunity. Besides the standard SR-IOV and VMware NetQueue support, the X3100 NICs also have a proprietary SR-IOV implementation.
Poprietary solutions only make sense if they offer enough advantages. Neterion claims that the NIC chip has extensive hardware support for network prioritization and quality of service. That hardware support should be superior to hypervisor traffic shaping, especially on the receive side. After all, if bursty traffic causes the NIC to drop packets on the receive side, there is nothing the hypervisor can do: it never saw those packets pass. To underline this, Neterion equips the X3100 with a massive 64MB receive buffer; for comparison, the competitors have in the best case a 512KB receive buffer. This huge receive buffer should ensure that the QoS is guaranteed even if relatively long bursts of network traffic occur.
Neterion NICs can be found in IBM, HP, Dell, Fujitsu, and Hitachi machines. Neterion is part of Exar and has also access to a world distributor channel. The typical price of this NIC is around $745.
The Competition: Solarflare
Solarflare is a relatively young company, founded in 2001. The main philosophy of Solarflare has been been “make Ethernet [over copper] better” (we added “over copper”). Solarflare NICs support optical media too, but Solarflare made headlines with their 10G Ethernet copper products. In 2006, Solarflare was the first with 10GBase-T PHY. 10GBase-T allows 10Gigabit over the very common and cheap CAT5E and the reasonably priced CAT6 and CAT6A UTP cables. Solarflare is also strongly advocating the use of 10GbE in the HPC world with the claim that the latency of 10GbE can be as low as 4 µs. In June of this year, Solarflare launched the SFN5121T, a dual-ported 10Gbase-T NIC which featured a very reasonable 12.9W power consumption, especially for a UTP based 10GbE product. In January of this year, the company decided to start selling NIC adapters directly to the end-user.
As we got an SFP+ Neterion X3120, we took a look at the optical brother of the SFN5121T, the SFN5122F. Both Solarflare NICs support SR-IOV and and make use of PCIe 2.0. The SFP+ SFN5122F should only consume a very low 4.9W for the complete card. Solarflare pulled this off by designing their own PHYs and reducing the chip count on the NIC. Although our power measurement methods (measured at the wall) are too crude to measure the exact power consumption, we can confirm that the Solarflare NIC consumed the least of the three NICs we tested.
The Solarflare chips are only slightly more expensive than the other NICs. Prices on the web were typically around $815.
The oldies
It is always interesting to get some “historical perspective”. Do these new cards outperform the older ones by a large margin? We included the Neterion XFrame-E for one test, and used the multi-queue pioneer and the Intel 82598, the 10GbE price breaker, as the historical reference for every benchmark. We'll try to add the Intel 82599 which also support SR-IOV, has a larger receive buffer, more queues and is priced around $700. We plugged those NICs in our Supermicro Twin² for testing.
Facebook
Twitter
RSS
38 Comments
View All Comments
blosphere - Wednesday, November 24, 2010 - link
Oh my cable arms on the first page pic :(And about the consolidation, you don't want to do it that way. The proper way is to have two 1-port 10g cards or if you're counting every dollar, one 2-port card. Then you set the production traffic to active/standby config (different vlans of course) and when configuring the vmotion/vkernel port you go and override the port failover order to reverse the port priority from the production traffic (own vlans of course).
This way you utilise both ports on the cards and you have mediocre HA (not that vmware should be called a HA system in the first place) since the production would failover to the vmotion/vkernel port and vice versa.
All this stuff is in the vmware/cisco whitepaper. Deployed already a few years ago to our datacentres worldwide, around 100 esxi hosts and 3000+ vm guests, works like charm when things start going wrong. Of course vmware itself does cause some problems in a port loss situation but that's a different story.
mino - Wednesday, November 24, 2010 - link
Agreed, Agreed and again Agreed :).Dadofamunky - Thursday, November 25, 2010 - link
Two thumbs up for this.DukeN - Wednesday, November 24, 2010 - link
And what type of switch would actually have the switching capacity to push this type of traffic through in a dedicated manner? That is a cost to be considered.That being said, I think well priced FC might still be better from a CPU usage standpoint.
mino - Wednesday, November 24, 2010 - link
FC is better at everything! Problem being, it is a "bit" more expensive.So for an SMB or storage IO light apps? 10G all the way.
For an enterprise database stuff? Think about it very thouroughly before commiting to 10G. And even then,you better forget about iSCSI.
Consolidating everything-ethernet info 2*10G ? Great. Just do it!
But do not forget to get security boys on-board before making a proposal to your CIO :D
No, even Nexus 1000V would not help you ex-post ...
Inspector2211 - Wednesday, November 24, 2010 - link
Myricom was one of the 10G pioneers and now has a 2nd generation lineup of 10G NICs, with any phsyical connection option you can imagine (thick copper, thin copper, long range fiber, short range fiber).I picked up a pair of new first-gen Myricom NICs on eBay for $200 each and will conduct my own performance measurements soon (Linux box to Linux box).
iamkyle - Wednesday, November 24, 2010 - link
Last I checked, Myricom has no 10G over CAT5e/6 UTP product available.mianmian - Wednesday, November 24, 2010 - link
I guess the lightpeak products May first hit the 10G Ethernet market. it will greatly reduce the cost&energy for those servers.mino - Wednesday, November 24, 2010 - link
First:There is not mentioned in the article what kind of setup you are simulating.
Surely the network(HTTP ?) latency is not in tens of milliseconds, is it ?
Second:
Port consolidation? Yes, a great thing, but do not compare oranges to apples!
There is a huge difference in consolidating those 10+ Ethernet interfaces (easy) and joining in a previously FC SAN (VERY hard to do properly).
You are pretending that Ethernet (be it 1Gb or 10Gb) is in the performance class of even 4G FC SAN's is a BIG fail.
10Gb Ethernet SAN (dedicated!) is a great el-cheapo data streaming solution.
Rather try not hitting that with a write-through database.
If your 4G SAN utilization is in the <10% range and you have no storage-heavy apps, FCoE or even iSCSI is a very cost-effective proposition.
Yet even then it is prudent to go for a 2*10G + 2*10G arrangement of SAN + everything else.
I have yet to see a shaper who does not kill latency ...
Provided no test description was given, one has to assume you got ~4x the latency when shaping as well.
The article on itself was enlightening so keep up the good work!
Please, try not thinking purely SMB terms. There are MANY apps which would suffer tremendously going from FC latency to Ethernet latency.
FYI, One unnamed storage virtualization vendor has FC I/O operation pass-through-virtualization-box capability of well under 150us.
That same vendor has observed the best 1GbE solutions choke at <5k IOps, 10GbE at ~10k IOps while a basic 2G FC does ~20k IOps, 4G ~40k IOps and 8G up to ~70k IOps.
JohanAnandtech - Thursday, November 25, 2010 - link
I agree with you that consolidating storage en network traffic should not be done on heavy transaction databases that already require 50% of your 10 GbE pipe.However, this claim is a bit weird:
"That same vendor has observed the best 1GbE solutions choke at <5k IOps, 10GbE at ~10k IOps while a basic 2G FC does ~20k IOps, 4G ~40k IOps and 8G up to ~70k IOps."
Let us assume that the average block size is 16 KB. That is 5000x16 KB or 80 MB/s for the 1 G solution. I can perfectly live with that claim, it seems very close to what we measure. However, claiming that 10G ethernet can only do twice as much seems to indicate that the 10G solution was badly configured.
I agree that the latency of FC is quite a bit lower. But let us put this perspective: those FC HBA have been communicating with disk arrays that have several (in some cases >10) ms of latency in case of write-through database. So 150us or 600us latency in the HBA + cabling is not going to make the difference IMHO.
To illustrate my point: the latency of our mixed test (Iometer/IxChariot) is as follows: 2.1 ms for the disktest (Iometer 64 KB sequential), 330 us for the networktest (high performance script of IxChariot). I think that is very acceptable to any application.