Broadband network protocol overhead

I have an assignment for my university course that requires to review the 2020 report of Measuring Broadband Australia that covers the current state of the connection quiality within Australian NBN broadband network.

In particular in one of the sections the report covers the topic of why most of the users of NBN can usually attain between 90 and 95% of advertised download speed and the authors say:

This reiterates the point raised in previous reports that NBN tier speeds are not provisioned so that maximum plan speeds are attainable after accounting for protocol overhead.

The go on to further elaborate in the footnote that:

Protocol overhead include packet headers, which are added to network communications to ensure that they arrive at the right network address. Packet headers take up space, which means that the connection has less room for whatever data is being sent.

This gap of 5-10% of the advertised speed appears to be independent of the advertised speed so a person on 50/20 plan loses 2.5-5 Mbps and on 100/40 plan they lose 5-10 Mbps and this seems like an extremely high overhead.

Unfortunately, I couldn't find any reliable sources on protocol overhead for broadband connections and was wondering if anyone here knows any sources or has any information.

For information, the Australian NBN is a combination of fibre-to-the-curb (FTTC), fibre-to-the-node (FTTN), fibre-to-the-premises (FTTP) and hybrid fibre coaxial (HFC).

As a starting point, most Internet connections are set up to carry 1500-byte IP packets. (This is practically never higher, as IP packets and especially TCP connections will end up using the smallest MTU along the whole path anyway.)

Around the 1500 byte boundary, there is both external overhead (which causes the whole message to grow) and internal overhead (which causes the available data space to shrink). Both of those contribute equally to the total "wasted" throughput. Note that those aren't official terms, I just made them up on the spot.

External overhead depends on the underlying connection type (i.e. probably what you're referring to as "broadband network protocol"):

• Ethernet puts each IP packet in an Ethernet frame with a 14-byte header and a 4-byte checksum (plus a 4-byte 802.1Q tag if VLANs are being used).

• VDSL usually works in PTM mode, which carries Ethernet frames but with an additional 4-byte PTM overhead.

• GPON also carries Ethernet frames, adding its own overhead which I don't yet understand (and things seem to work quite differently in 'up' and 'down' directions).

  It also shares the physical link between multiple customers (not sure how many is common, but 32 seems like a possible number).

• ADSL divides everything into 53-byte ATM cells (5-byte header and 48-byte payload), so in the usual "Ethernet emulation" mode you need 32 cells to send a full-size 1500-byte IP packet, and the cell headers add up to 160 bytes (11% overhead!).

This means a full-size 1500-byte IP packet takes at least 1518 bytes to send.

While the internal overheads are nearly always the same regardless of connection type (i.e. whether it's fiber or dial-up, they all eat into the 1500-byte MTU):

• If you have to use PPPoE, it shrinks the available MTU by 8 bytes.

• The IP packet header itself takes up 20 or 40 bytes (in IPv4 and IPv6 respectively).

• TCP-based connections need 20 bytes per packet for the TCP segment header (and often 12 more for the 'Timestamp' option).

• When downloading over HTTPS, each TCP segment carries at least one TLS record with its 5-byte header and a 24-byte MAC (message authentication code). The latter is for AES-GCM but varies per cipher suite.

This means that a full-size 1500-byte IP packet (assuming no PPPoE) typically holds between 1400 and 1460 bytes of useful data.

So in total, assuming Ethernet, in the best case (plaintext over IPv4) approx. 4.5% of the total frame size consists of overhead, and in the worst case (TLS over IPv6 in a VLAN) it's approx 8%. With DSL or GPON it only gets worse.

This is why some programs and some people use the conversion "1 MB/s ≈ 10 Mbps" (instead of the technically correct 8 Mbps), as they show the useful throughput in MB/s but total throughput in Mbps.

I recommend reading user1686's answer first for a general overview. I'll be going into a bit more detail on the NBN network.

Since we're talking about the NBN specifically, I'll be pulling from their Wholesale Broadband Agreement, especially the nbn™ Ethernet - Product Description 4.1, updated May 2021 and nbn™ Ethernet - Product Technical Specification 4.1, updated May 2021.

NBN basic background

First, let's start with the Product Description to get some basic information:

The nbn™ Ethernet Product ... is an Ethernet-based Layer 2 virtual connection that carries traffic between a UNI used to serve a Premises and a POI

An Access Virtual Circuit or AVC is an Ethernet-based Layer 2 virtual connection on the Fibre Network, FTTB Network, FTTN Network, FTTC Network, HFC Network, Wireless Network or Satellite Network that carries RSP traffic to and from a UNI used to serve a Premises.

The User Network Interface or UNI is a physical port to which nbn supplies nbn™ Ethernet in respect of a Premises.

NBN Information Rate (Ethernet frames)

Then the Product Technical Specification has "§ 2.2.2 Bandwidth Profile Parameter Considerations", which describes how information rate limitations are applied:

All Information Rate limitations, including as set out in this nbn™ Ethernet Product Technical Specification, are enforced at the NNI interface between the RSP and the nbn™ Network.

Where the bandwidth profile is equivalent to or greater than the negotiated Line Rate, a degraded useable payload will occur.

The Peak Information Rate for nbn™ Ethernet is calculated on Layer 2 Ethernet service frames, over the series of bytes from the first bit of the Destination MAC Address through the last bit of the Frame Check Sequence. IEEE 802.3 physical-layer fields such as the preamble, start of frame delimiter and inter-frame gap are not included in the Bandwidth Profile.

Unfortunately, further detail of what exactly is in those Ethernet frames is in separate Network Interface Specification documents that do not appear to be publicly available. If we step back to the NBN Co Ethernet Bitstream Service Product Technical Specification 2.12, updated July 2017, no longer in force, we can get some insight before they removed them from this document:

§ 3.1.5 AVC/CVC Service Addressing Mode A uses a two-level VLAN addressing scheme at the NNI, which is compliant with IEEE802.1ad (Provider Bridges) to identify individual 1:1 AVC and CVC services.

The IEEE802.1ad S/C-TAG adds an additional 4 bytes overhead each (total of 8) to the Ethernet frame. Of course, we are making the assumption that this is still the case.

Ethernet payload additional overheads

This more or less follows the "internal overheads" as described in user1686's answer. Note that whether PPPoE is used depends on the RSP; the NBN does not care what goes in the payload.

Additional line rate limitations

Aside from the AVC limits, there can also be limitations due to the User Network Interface (UNI), i.e. the specific technology used to connect the user to the NBN.

These are generally limited by what is achievable, and any protocol overheads are not included in the information rate limitations (with maybe an exception for VDSL2 Ethernet over copper? see below). Specifically, any DSL overheads are not included in the advertised bandwidth nor checked by any shaping or policing -- but, of course, if your line rate is too low it will impact your bandwidth before any shaping or policing come into effect.

Note that there are two data categories of UNI:

• UNI-D, covering the networks that provide an Ethernet port to the end user (incl. FTTP, FTTC, HFC, FW, Satellite). This provides a 10/100/1000BASE-T/TX port. On UNI-D, a user is generally limited by AVC only, unless they purchased a 1000Mbps AVC where they will be limited by the port speed (1000BASE-T for the entire Ethernet frame).

  Note that there can be additional limitations on achievable bandwidth depending on the network. This is most apparent on Satellite and FW, followed by HFC.

• UNI-DSL, covering those networks that use VDSL2 and require a VDSL2 modem (FTTN, FTTB). A user can be limited by the line rate, which depends on the quality of the copper phone wiring and distance to the node.

  § 3.3.2.2 For DSL services the Information Rate is limited to the lesser of the aggregate AVC bandwidth and the actual Line Rate on the UNI-DSL. Note also that for VDSL2 the Line Rate and Information Rate are subject to VDSL2 Ethernet over copper framing overheads as defined in the ITU-T VDSL2 specification G.993.2.

• UNI-V is that is a voice-only product with very limited high-priority bandwidth (for VoIP) that is not available outside of FTTP and not supported by all RSPs. The UNI-V port is an ATA and provides only an analog phone connection to the user (VoIP/SIP is handled on the NBN/RSP side and not exposed).

A comment on overprovisioning

Pre-mid-2020, NBN policed traffic at the advertised AVC speeds, which resulted in lower observed speeds on common internet speed tests (e.g. speedtest.net, which uses either WebSockets or HTTP depending on browser). user1686 has already calculated these overheads as, roughly, 4.5-8%.

Around mid-2020, NBN started overprovisioning the downstream. Some documents from around that time describe this as:

For Home Fast and Home Superfast, nbn network management policers will provide an additional 15% overhead allowance to the AVC TC-4 downstream PIR at layer 2. This will allow End Users to potentially achieve a downstream speed on their network approximate to numerical value of the published layer 2 wholesale downstream PIR where nbn infrastructure has capacity.

This 15% overprovision is enough to ensure that common speed tests (over WSS/HTTP[S] payloads) should never show less than the advertised downstream AVC rates due to NBN traffic policing.