Upgrading to Gigabit Network - Enabling Jumbo Frames

I would like to begin to upgrade my SOHO network to gigabit (from 10/100) and have heard a bit on Jumbo Frames.

What would be the best way to implement Jumbo Frames on a network? From what I can tell in order for it to work properly all network gear on the network must support Jumbo Frames. Is this true?

If I have specific gear (e.g. network printer) that cannot be updated to GB ethernet will this prevent me from enabling Jumbo Frames?

What are some of the gotcha's of enabling Jumbo Frames?

First, it might be best to explain what jumbo frame ethernet is. Ethernet is a layer 2 networking technology and its Protocol Data Unit (PDU) is a frame. For reference, an L3PDU (IP layer) is a packet, and an L4PDU (tcp/udp) is a segment.

An ethernet frame (there are several types of ethernet but we can generalize here) consists of a header (containing, among other things, a source MAC, a destination MAC, an 802.1q VLAN tag, etc) the data, or paylod, of the frame, and a CRC checksum used to validate the successful transmission of the frame.

The original ethernet specified a frame size (the valume of data in the entire frame, including the header and checksum) as 1500 bytes (or possibly 1518, have to look it up). This number struck a balance between amount of data to send at once and the likelihood of that transmission failing or colliding and having to be retransmitted. With the advent of fast, full duplex LANs, people realized that performance could be improved by increasing the ethernet frame size. The traditional size of jumbo frames are 9000 bytes per frame, though this is mostly convention.

On a rock solid, full duplex LAN (or VLAN) in which all elements are expecting to receive jumbo frame ethernet, it actually does improve performance. The problem with this scenario is if you introduce a network element or end device which doesn't expect it. In the best case, it will result in a performance degradation as packets are lost because receiving devices are expecting only 1518 bytes in a frame.

Now to your specific questions:

What would be the best way to implement Jumbo Frames on a network?

This is a subjective question. At my place of business we chose to implement it only where we knew we had all the variable under control and we knew it would help. To do this we implemented it in a special "private" vlan that only specific devices could access via their second NICs. Specifically, we put the second NIC of our file servers and application servers into this new VLAN and then changed all references to the IP scheme used in this VLAN. That allows us to narrowly target (no ones going to plug a desktop machine into this VLAN) the specific area we know would most benefit (the highest utilization data links in our infrastructure). This maximizes gain while minimizing risk.

More specifically, on the network side (using IOS), we built VLANs dedicated to the jumbo frame devices, then added "mtu 9000" to their vlan definition. Every interface on the switch that would be using this network was put in this vlan using something like "switchport access vlan 11". On the linux machines (which have eth0 connected to the standard network and eth1 connected to the jumbo frame network) we added "MTU=9000" to /etc/sysconfig/network-scripts/ifcfg-eth1. Because we never route these packets (it's impossible for anything not directly connected to the jumbo frame VLAN to speak with a NIC on the jumbo frame VLAN) we never had to worry about a router configuration.

From what I can tell in order for it to work properly all network gear on the network must support Jumbo Frames. Is this true?

Yeah, pretty much. All network "clients" (by which I means servers/desktops/IPKVMs/IP environmental monitors, etc) must speak it also or, as mentioned above, you'll have lots of semi-reachable machines (they'll ping, and any L3 or L4PDU that's less than 1500 bytes will succeed, which means, as an example, your mail server will ping, and you'll be able to hand deliver what will likely be a small test message. But when you try to deliver a real mail (the one with the excel attachment that is pushed the frame size >1500 bytes) it will mysteriously fail).

If I have specific gear (e.g. network printer) that cannot be updated to GB ethernet will this prevent me from enabling Jumbo Frames?

If that's the case, here's what I would do (assuming network gear that can handle this):

• build two VLANs, one with jumbo frame and one without
• assign all your network devices to one vlan or the other
• in your router, and switches, implement the jumbo frame vlan and change the frame size on any network clients.

This means that you will no longer have a flat L2 topology on your network. For instance, if from your jumbo-frame enabled server you want to print to your non-jumbo frame printer, the packets will have to be routed (travel through your router, the frames rewritten into a more conventional size, and then sent to the printer on the other VLAN). This means that communication between your jumbo frame and non-jumbo frame machines will be slightly poorer than it was before, but the data transfer rates between all devices on the jumbro frame VLAN will be better. It's really just a judgement call.

What are some of the gotcha's of enabling Jumbo Frames?

Hopefully covered above. Good luck!

You may find Jeff Atwood's post on Jumbo Frames informative.

Highlights of the post:

• 20% Performance Boost
• For a large frame to stay intact, every device it passes through must support that frame size
• Switches that don't support Jumbo Frames will drop them

You can use ping.exe to check the max size of the packets and compare that to your Jumbo Frames settings.

ping -l 4096 -f server

Adjust the packetsize used by -l, and use -f to set the DO_ NOT_FRAGMENT flag. When you reach your maximum packetsize you'll get a "Packet needs to be fragmented but DF set".

That will give you an indication if Jumbo Frames works or not.

Yes, everything must support Jumbo Frames - treat it like switching betwen token ring and ethernet. The only difference is that some devices might appear to still work for a short time or intermittently - this can also be a major headache if you don't keep track of which devices you reconfigured on a large network (i.e. 2 weeks later you get a trouble ticket from some user with a printer stuffed in the back of their cubicle that "just now" stopped working). The same applies with any new stuff - you'll need to setup a procedure to reconfigure any new devices and computers with jumbo frames, to avoid support calls when they don't work beyond the initial bootup.