HP access points sending out multicast packets from IPs that aren't their own
I'm not our normal network guy... I've just been drafted to help with this issue, so please bear with me.
We have a fairly large (~4,000 devices?) network mostly comprised of HP Procurve gear. From time to time over the last several weeks, we've been getting some broadcast storms that pretty much keep all other traffic from being sent across the network. I set up Wireshark to do 5MB dumps, and I caught some of this in the act this morning.
You can download the packet capture. The fun began at packet #23968. A seemingly malformed NBNS packet gets repeated over and over again. However, it isn't just a straight loop. The source (143.226.8.185) and destination (143.226.44.79) IP addresses stay the same, but the source MAC address changes. The first packet seems to come from some insignificant device on the network, and is sent to a multicast address, 01:00:5e:7f:ff:fa. All of the packets after that come from the MAC addresses of our HP wireless access points and are sent to a different multicast address, 01:00:5e:62:2c:4f.
Here is the first packet:
No. Time Source Destination Protocol Info
23968 122.229240 143.226.8.185 143.226.44.79 NBNS Unknown operation (10) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding)[Malformed Packet]
Frame 23968 (1038 bytes on wire, 1038 bytes captured)
Arrival Time: Sep 15, 2010 08:32:44.329966000
[Time delta from previous captured frame: 0.004744000 seconds]
[Time delta from previous displayed frame: 0.004744000 seconds]
[Time since reference or first frame: 122.229240000 seconds]
Frame Number: 23968
Frame Length: 1038 bytes
Capture Length: 1038 bytes
[Frame is marked: True]
[Protocols in frame: eth:ip:udp:nbns]
[Coloring Rule Name: SMB]
[Coloring Rule String: smb || nbss || nbns || nbipx || ipxsap || netbios]
Ethernet II, Src: IntelCor_d2:5e:6b (00:1f:3b:d2:5e:6b), Dst: IPv4mcast_7f:ff:fa (01:00:5e:7f:ff:fa)
Destination: IPv4mcast_7f:ff:fa (01:00:5e:7f:ff:fa)
Address: IPv4mcast_7f:ff:fa (01:00:5e:7f:ff:fa)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: IntelCor_d2:5e:6b (00:1f:3b:d2:5e:6b)
Address: IntelCor_d2:5e:6b (00:1f:3b:d2:5e:6b)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Type: IP (0x0800)
Trailer: 7773643D22687474703A2F2F736368656D61732E786D6C73...
Frame check sequence: 0x6f70653e [incorrect, should be 0x30019938]
Internet Protocol, Src: 143.226.8.185 (143.226.8.185), Dst: 143.226.44.79 (143.226.44.79)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 203
Identification: 0x00d0 (208)
Flags: 0x00
0.. = Reserved bit: Not Set
.0. = Don't fragment: Not Set
..0 = More fragments: Not Set
Fragment offset: 0
Time to live: 128
Protocol: UDP (0x11)
Header checksum: 0xe485 [correct]
[Good: True]
[Bad : False]
Source: 143.226.8.185 (143.226.8.185)
Destination: 143.226.44.79 (143.226.44.79)
User Datagram Protocol, Src Port: netbios-ns (137), Dst Port: netbios-ns (137)
Source port: netbios-ns (137)
Destination port: netbios-ns (137)
Length: 183
Checksum: 0x01db [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
NetBIOS Name Service
Transaction ID: 0x4d2d
Flags: 0x5345 (Unknown operation)
0... .... .... .... = Response: Message is a query
.101 0... .... .... = Opcode: Unknown (10)
.... ..1. .... .... = Truncated: Message is truncated
.... ...1 .... .... = Recursion desired: Do query recursively
.... .... ...0 .... = Broadcast: Not a broadcast packet
Questions: 16722
Answer RRs: 17224
Authority RRs: 8234
Additional RRs: 8264
Queries
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (12081)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (12081)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (11631)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (11631)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25701)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (25701)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25914)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (25914)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25970)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (25970)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (18273)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (18273)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (24953)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (24953)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (26979)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (26979)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (3338)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (3338)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (14882)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (14882)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (28730)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (28730)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25455)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (25455)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (8717)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (8717)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (28513)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (28513)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (29287)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (29287)
[Malformed Packet: NBNS]
[Expert Info (Error/Malformed): Malformed Packet (Exception occurred)]
[Message: Malformed Packet (Exception occurred)]
[Severity level: Error]
[Group: Malformed]
Here is the next packet:
No. Time Source Destination Protocol Info
23969 122.229836 143.226.8.185 143.226.44.79 NBNS Unknown operation (10) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding) unknown Illegal NetBIOS name (1st character not between A and Z in first-level encoding)[Malformed Packet]
Frame 23969 (217 bytes on wire, 217 bytes captured)
Arrival Time: Sep 15, 2010 08:32:44.330562000
[Time delta from previous captured frame: 0.000596000 seconds]
[Time delta from previous displayed frame: 0.000596000 seconds]
[Time since reference or first frame: 122.229836000 seconds]
Frame Number: 23969
Frame Length: 217 bytes
Capture Length: 217 bytes
[Frame is marked: True]
[Protocols in frame: eth:ip:udp:nbns]
[Coloring Rule Name: SMB]
[Coloring Rule String: smb || nbss || nbns || nbipx || ipxsap || netbios]
Ethernet II, Src: HewlettP_05:de:da (00:17:a4:05:de:da), Dst: IPv4mcast_62:2c:4f (01:00:5e:62:2c:4f)
Destination: IPv4mcast_62:2c:4f (01:00:5e:62:2c:4f)
Address: IPv4mcast_62:2c:4f (01:00:5e:62:2c:4f)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: HewlettP_05:de:da (00:17:a4:05:de:da)
Address: HewlettP_05:de:da (00:17:a4:05:de:da)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Type: IP (0x0800)
Internet Protocol, Src: 143.226.8.185 (143.226.8.185), Dst: 143.226.44.79 (143.226.44.79)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 203
Identification: 0x00d0 (208)
Flags: 0x00
0.. = Reserved bit: Not Set
.0. = Don't fragment: Not Set
..0 = More fragments: Not Set
Fragment offset: 0
Time to live: 127
Protocol: UDP (0x11)
Header checksum: 0xe585 [correct]
[Good: True]
[Bad : False]
Source: 143.226.8.185 (143.226.8.185)
Destination: 143.226.44.79 (143.226.44.79)
User Datagram Protocol, Src Port: netbios-ns (137), Dst Port: netbios-ns (137)
Source port: netbios-ns (137)
Destination port: netbios-ns (137)
Length: 183
Checksum: 0x01db [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
NetBIOS Name Service
Transaction ID: 0x4d2d
Flags: 0x5345 (Unknown operation)
0... .... .... .... = Response: Message is a query
.101 0... .... .... = Opcode: Unknown (10)
.... ..1. .... .... = Truncated: Message is truncated
.... ...1 .... .... = Recursion desired: Do query recursively
.... .... ...0 .... = Broadcast: Not a broadcast packet
Questions: 16722
Answer RRs: 17224
Authority RRs: 8234
Additional RRs: 8264
Queries
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (12081)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (12081)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (11631)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (11631)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25701)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (25701)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25914)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (25914)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25970)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (25970)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (18273)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (18273)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (24953)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (24953)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (26979)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (26979)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (3338)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (3338)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (14882)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (14882)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (28730)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (28730)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (25455)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (25455)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (8717)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (8717)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (28513)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (28513)
Illegal NetBIOS name (1st character not between A and Z in first-level encoding): type unknown, class Unknown (29287)
Name: Illegal NetBIOS name (1st character not between A and Z in first-level encoding)
Type: unknown
Class: Unknown (29287)
[Malformed Packet: NBNS]
[Expert Info (Error/Malformed): Malformed Packet (Exception occurred)]
[Message: Malformed Packet (Exception occurred)]
[Severity level: Error]
[Group: Malformed]
Crazy, no? If you look through the packet capture, you will see much of this packet repeated after that point. It goes on and on after that, in to several more files.
If this were a loop, then why would only our APs be sending this packet around? These APs are scattered all over our campus.
A bit more info about our network... It is all flat. Straight Ethernet runs to everything, and we have a class B block of IPs. No subnets. There is a packet shaper, firewall, and router between our network and our WAN connection.
Finally, if you see this post and it seems familiar to you, that is because I have posted a similar issue in the past that we still have yet to resolve, but haven't seen recently. That can be found at HP switches sending multi-cast ping requests.
Thank you much for your time!
Edit: Packet 23968 is confirmed to be the trigger of this multicast storm. I've replayed that one packet into our network and kicked it off again.
Edit/Update: Doing some more experimentation. I've taken one of our HP access points and plugged it in directly to my PC. Nothing else attached to the segment. If I replay the initial packet that was causing the problems to the AP, the AP replies once. If I replay the AP's reply back to the AP, it replies again. Each time it does this, the TTL is lowered. What is happening here is that the APs on the network initially hear the broken multicast packet from the host and reply to it over multicast. Each AP hears these replies from all of the other APs and it replies to them. Each AP hears all of the replies to the replies and replies to them. Fortunately, it lowers the TTL each time so the storm goes away as soon as the TTL hits 0, and the packet is killed. Now all I need to do is figure out how to stop this behavior!
The AP I have in front of me is an HP Procruve 420 J8130B.
Edit (SOLVED!): After trying seemingly every configuration setting on the AP, I was still unable to keep it from retransmitting those multicast packets. I discovered we weren't on the latest firmware, so I tried upgrading, but the problem persisted. Then I tried downgrading to version 2.1.7 from November 29, 2006. No problems with this firmware! APs running 2.1.7 do not retransmit the packet!!! I am still waiting to figure out how the junk data got on the network in the first place, but the problem is solved for now. We are making a bug report with HP.
Solution 1:
First and foremost, those are not NBNS packets, they're actually Universal Plug-n-Play packets attempting to search for "Internet Gateway Device" enabled devices. UPNP-IGD uses IPv4 multicast to locate such edge devices. The protocol, such as it is, says there should be only one. The give-away is in the packet payload:
M-SEARCH * HTTP/1.1 Host:239.255.255.250:1900 ST:urn:schemas-upnp-org:device:InternetGatewayDevice:1 Man:"ssdp:discover" MX:3 .xmlsoap.org/ws/2004/08/addressing" xmlns:
IGD is used by some applications to tell consumer NAT gateways how to handle NAT traversal for certain protocols. IM applications and the like. You can make Wireshark show things better by telling it to decode UDP/137 as HTTP for that capture.
Now, why this is causing a multicast storm is the big question. You're getting the same kind of packet well before the storm hits, but they're correctly being sent to 239.255.255.250:1900. Packet 23955, in fact, comes from the same device that starts the storm in 23968. However, packet 23968 shows the same destination MAC address (one indicating IPv4 Multicast) but has a destination IP address that is in your /16 block and should NOT be multicast.
Packet 23604 is also very malformed. It has a valid Ethernet header, but the IP header is strangely truncated and ends in the same UPNP-IGD string I quoted above. The device that issued this strange, strange packet is the same device (well, comes from the same MAC address anyway) as packet 23968 that kicked off the multicast storm.
My best bet at this point is that the device at 00:1F:3B:D2:5E:6D is hosed in some way or is uniquely not handling these UPNP search requests correctly. Packet 24717 shows another M-SEARCH request going to 239.255.255.250:3702 that also comes from the same device. Right IP address, wrong port (should be 1900).
My guess is that the multicast storm is being kicked off by a packet with a Unicast IP address arriving with a multicast MAC address, and your network devices not handling that invalid case correctly. This is suggestive in the fact that the packets after the initial one all claim to source from the same IP (143.226.8.185), but the MAC address are all different. You have a bad device that managed to find a bug in multicast/unicast handling of your net-devices.