Hubs/switches taking out switches?
Here's the issue...we have a network with a lot of Cisco switches.
Someone plugged in a hub on the network, and then we started seeing "weird" behavior; errors in communication between clients and servers, or network timeouts, dropping network connections, etc. It seemed that somehow that hub (or SOHO switch) was particularly freaking out our Cisco 3700 series switches.
Disconnect that hub or netgear-type SOHO switch and things settled down again.
We're in the process of trying to get a centralized logging server for SNMP and management, etc., to see if we can trap errors or narrow down when someone does this sort of thing without our knowledge because things seem to work, for the most part, without issue, we just get freaky oddball incidents on particular switches that don't seem to have any explanation until we find out someone decided to take matters into their own hands to expand available ports in their room.
Without getting into procedure changes or locking down ports or "in our organization they'd be fired" answers, can someone explain why adding a small switch or hub, not necessarily a SOHO router (even a dumb hub apparently caused the 3700's to freak out) sending DHCP request out, will cause issues? The boss said it's because the Cisco's are getting confused because that rogue hub/switch is bridging multiple MAC's/IP's into one port on the Cisco switches and they just choke on that, but I thought their routing tables should be able to handle multiple machines coming into the port. Anyone see that behavior before and have a clearer explanation of what's happening?
I'd like to know for future troubleshooting and better understanding that just waving my hand and saying "you just can't".
Here's a show run
Current configuration : 25591 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname ###########
!
boot-start-marker
boot-end-marker
!
enable secret 5 ############
!
!
!
no aaa new-model
switch 1 provision ws-c3750g-24ps
switch 2 provision ws-c3750-48ts
switch 3 provision ws-c3750-48ts
switch 4 provision ws-c3750-48ts
switch 5 provision ws-c3750-48ts
system mtu routing 1500
authentication mac-move permit
ip subnet-zero
ip routing
!
!
!
mls qos map policed-dscp 24 26 46 to 0
mls qos map cos-dscp 0 8 16 24 32 46 48 56
mls qos srr-queue input bandwidth 90 10
mls qos srr-queue input threshold 1 8 16
mls qos srr-queue input threshold 2 34 66
mls qos srr-queue input buffers 67 33
mls qos srr-queue input cos-map queue 1 threshold 2 1
mls qos srr-queue input cos-map queue 1 threshold 3 0
mls qos srr-queue input cos-map queue 2 threshold 1 2
mls qos srr-queue input cos-map queue 2 threshold 2 4 6 7
mls qos srr-queue input cos-map queue 2 threshold 3 3 5
mls qos srr-queue input dscp-map queue 1 threshold 2 9 10 11 12 13 14 15
mls qos srr-queue input dscp-map queue 1 threshold 3 0 1 2 3 4 5 6 7
mls qos srr-queue input dscp-map queue 1 threshold 3 32
mls qos srr-queue input dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23
mls qos srr-queue input dscp-map queue 2 threshold 2 33 34 35 36 37 38 39 48
mls qos srr-queue input dscp-map queue 2 threshold 2 49 50 51 52 53 54 55 56
mls qos srr-queue input dscp-map queue 2 threshold 2 57 58 59 60 61 62 63
mls qos srr-queue input dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31
mls qos srr-queue input dscp-map queue 2 threshold 3 40 41 42 43 44 45 46 47
mls qos srr-queue output cos-map queue 1 threshold 3 5
mls qos srr-queue output cos-map queue 2 threshold 3 3 6 7
mls qos srr-queue output cos-map queue 3 threshold 3 2 4
mls qos srr-queue output cos-map queue 4 threshold 2 1
mls qos srr-queue output cos-map queue 4 threshold 3 0
mls qos srr-queue output dscp-map queue 1 threshold 3 40 41 42 43 44 45 46 47
mls qos srr-queue output dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31
mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55
mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63
mls qos srr-queue output dscp-map queue 3 threshold 3 16 17 18 19 20 21 22 23
mls qos srr-queue output dscp-map queue 3 threshold 3 32 33 34 35 36 37 38 39
mls qos srr-queue output dscp-map queue 4 threshold 1 8
mls qos srr-queue output dscp-map queue 4 threshold 2 9 10 11 12 13 14 15
mls qos srr-queue output dscp-map queue 4 threshold 3 0 1 2 3 4 5 6 7
mls qos queue-set output 1 threshold 1 138 138 92 138
mls qos queue-set output 1 threshold 2 138 138 92 400
mls qos queue-set output 1 threshold 3 36 77 100 318
mls qos queue-set output 1 threshold 4 20 50 67 400
mls qos queue-set output 2 threshold 1 149 149 100 149
mls qos queue-set output 2 threshold 2 118 118 100 235
mls qos queue-set output 2 threshold 3 41 68 100 272
mls qos queue-set output 2 threshold 4 42 72 100 242
mls qos queue-set output 1 buffers 10 10 26 54
mls qos queue-set output 2 buffers 16 6 17 61
mls qos
!
!
!
!
!
spanning-tree mode pvst
spanning-tree etherchannel guard misconfig
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
class-map match-all AutoQoS-VoIP-RTP-Trust
match ip dscp ef
class-map match-all AutoQoS-VoIP-Control-Trust
match ip dscp cs3 af31
!
!
policy-map AutoQoS-Police-CiscoPhone
class AutoQoS-VoIP-RTP-Trust
set dscp ef
police 320000 8000 exceed-action policed-dscp-transmit
class AutoQoS-VoIP-Control-Trust
set dscp cs3
police 32000 8000 exceed-action policed-dscp-transmit
!
!
!
!
interface GigabitEthernet1/0/1
switchport trunk encapsulation dot1q
switchport trunk native vlan 11
switchport mode trunk
spanning-tree portfast
!
!
!
interface GigabitEthernet5/0/1
!
interface GigabitEthernet5/0/2
!
interface GigabitEthernet5/0/3
!
interface GigabitEthernet5/0/4
!
interface Vlan1
ip address ############## 255.255.0.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 ##############
no ip http server
no ip http secure-server
!
!
ip sla enable reaction-alerts
!
!
!
line con 0
line vty 0 4
password 7 ############
login
line vty 5 15
password 7 ############
login
!
end
Solution 1:
We operate a couple network implementations where third-party connections are linked up to a centralized Cisco backbone (i.e. multi-tenant setup). I can say I've seen a bunch of diverse (okay, ghetto) devices connected up to the Catalyst platform, and if there's one thing I've learned, it's that the Cisco platform is remarkably resilient to these kinds of things.
There is one achilles heel, though - A cheap hub in the right configuration can easily bring down an entire network with a broadcast storm, and it's not even the Cisco platform's fault. I discovered this in a production configuration, and the only real research I did was finding the closest trash can for that hub, but here's how it happened:
- Connect hub to Cisco switch as normal, with uplink port
- Connect a workstation to a hub port (in our case, running Windows XP OS, but shouldn't matter)
- Connect two other ports together on the hub (either directly, with a single CAT5 or indirectly through another hub).
Everything runs smoothly until that workstation sends out a broadcast announcement. While the hub and the Cisco are smart enough to prevent a broadcast storm on other broadcast packets, the hub isn't smart enough to detect that two of its ports are connected to each other, and will use up almost 100% of its processing power to broadcast that packet in an infinite loop back and forth, as well as out all the other ports (i.e. the uplink to your Cisco).
If this is the case in your configuration, you will notice that across your network, all of the ports on that broadcast VLAN will go nuts, up until the hub can't sustain the capacity and drops the magical looping packet (could be a couple minutes depending on the competing traffic), and then all is back to normal.
In this situation SNMP won't help you since all the ports on that VLAN go crazy with traffic. However, Wireshark is your friend here, since it's easy to capture which IP (and sometimes machine name if it's a broadcast packet) caused the loop, and quickly locate the offending device.
May not be the exact case you're experiencing, but this one bit us hard and might give you some ideas to research with your situation.
Solution 2:
Trivial points, but I haven't seen them mentioned yet:
Make sure your switch ports aren't forced to full duplex because they won't work with hubs if they are. Come to think of it - if their duplex or their speeds are forced they likely won't connect correctly to a Netgear type switch which wants to auto-negotiate
Make sure your users don't connect the switch to two network outlets "to double their bandwidth".
Cisco switches used to (and maybe still do - I'm out of that business now) have a feature called "Faststart". For any port with Faststart enabled the switch would not do a full spanning tree analysis and by enabling the feature the admin "promised" not to connect a switch or hub to that port. The reason for doing this was to avoid the client PC's DHCP request timing out before the Cisco decided it was safe to enable the port. You may want to look at that also. (If I have totally mis-remembered all this I apologise in advance and hope someone with more up-to-date knowledge will correct me)