Why does multicasting make the WAN inaccessible?

I'm hesitant to ask this question as I may not be able to perform the diagnostic steps to find out what is up, but:

I need to re-image a lab in a school, possibly during school hours. I am using a multicast stream as it hardly makes sense for the server to share out one file 30 times. I adjust the speed way down so there would be headroom and the rest of the network should be reasonably unaffected.

What I found is that doing so appears to cut us off of the WAN. Other sites are inaccessible, as is the internet. The school can not be seen from elsewhere while the multicast restore is running. (The multicasting itself works fine).

I spoke to my coworker who is in charge of the networking when this happened at a different site, and he wasn't able to find out anything regarding what we were doing that would cause problems.

Details: I am doing an ASR multicast restore using DeployStudio on a Mac OS X Server. Most of our switches are Cisco gear. We do have a number of Wireless Access Points (and they seem to pass the multicast stream on wireless; not optimal, but not necessarily a problem.) I put the stream at a rate of 2 Mb/s (I think that is megabits, not bytes); systems restore fine at 8 and sometimes even 12 Mb/s (so I know there is head room).

It is my preference to do multicast streaming on a network physically separate from one in production use, or to do it over the summer when few staff members are around, but I don't have that luxury here.

What might account for this problem?

The problem is probably that your switch doesn't handle correctly multicast groups and then send multicast as broadcast (and so flood your wan link, or at least the wan router) with useless data).
So you must have at least one switch acting as an IGMP Querier, this can be enabled on Cisco 2960 and 3750 with the command ip igmp snooping querier. If your network only has 2950 you will definitively have problem.
You can easily see if your switch are broadcasting the multicast by running wireshark on a PC while you reinstall image on others computers. (you should not see the multicast data if all is working fine)

Please also tell us what multicast IP is used, some are reserved and should not be used, some can be routed and some are not routable (see http://iana.org/assignments/multicast-addresses/multicast-addresses.xml)

Flooding the router that handles your gateway access, perhaps?

Is there a way for you to get SNMP info from the router or get logging information from it, CPU status, etc.?

How big is the WAN connection? Depending on the multicast address being used it could be that the multicast traffic is saturating the WAN interface, for instance multicast address 224.0.0.1 means "all hosts on this subnet", which means the WAN interface has to listen for and then discard the multicast traffic.

If the WAN interface has to listen for and then discard the multicast traffic, and the multicast traffic is flowing at a rate of 8 to 12 Mbps, and the WAN link is smaller than 8 to 12 mbps then I could see this causing the problem.

My guess would be that it's not a WAN problem per-se, but that the LAN side interface of your router is getting swamped by multicast frames flooded by your switch.

As has been mentioned in another comment, you'll need to enable IGMP snooping to enable your switch to properly constrain multicast frames. You'll also probably need to enable the IGMP snooping querier, unless you've got a multicast (PIM) router on each of your VLAN's. On a Cisco switch, you can enable IGMP snooping and the snooping querier by entering the following two commands in global config mode:

ip igmp snooping
ip igmp snooping querier

You should ensure igmp snooping is enabled on every switch in your network. The snooping querier only needs to be enabled on one switch, assuming that switch has an IP in each of your VLAN's. My understanding is it won't hurt though to enable the snooping querier on every switch. Note that for the snooping querier to work, your switch will need an IP in each of your VLAN's, or at least each VLAN that has multicast traffic you're concerned about constraining.

In case you're curious about why you need IGMP snooping:

As you probably know, normally a switch delivers traffic by consulting it's CAM table. The CAM table is populated by inspecting the source MAC address of every frame received by the switch. Every source MAC address the switch sees is added to the CAM table, along with the switch port that the frame in question entered the switch on. In this way the switch "learns" what MAC addresses are connected to each port.

The switch uses the CAM table to determine where to deliver incoming frames. If the destination MAC address is found in the CAM table, then the switch knows which port to deliver the frame to, and the frame is delivered to only that port. If the destination MAC address is NOT found in the CAM table, the frame is flooded out to every port on the switch.

With multicast traffic, the the source MAC address of the frame will be the MAC address of the multicast sender, but destination MAC address of the frame will be the MAC address of a multicast group, not the MAC address of any specific individual PC. This multicast MAC address should never normally be the source address of any frame, so in normal operation the switch will never know where to send multicast frames. It will have no choice but to flood the frames out of every port. When this happens with a really large multicast stream, these flooded frames can sometimes overwhelm other systems on the network.

IGMP is actually a layer 3 protocol designed to allow IP hosts to inform IP routers that they would like to join a multicast group. Technically IGMP has nothing to do switches and layer 2 operation, but a number of switch vendors, including Cisco, have added features to their switches that allow the switch to listen in (or snoop) on the IGMP traffic between IP hosts and multicast enabled IP routers.

Unfortunately IGMP snooping only works if there is a multicast router on the subnet(s) in question. If there's no multicast enabled router, then there's no IGMP conversation to "snoop" on. That's where the IGMP snooping querier comes in. It send out the IGMP membership queries that would normally be sent by a PIM multicast router, thereby initiating the exchange for the switch to "snoop".

It would be nice if IGMP snooping was enabled by default on most switches, but I suppose the reason it isn't is because, while IGMP is an IETF standard, there's no actual standard for IGMP snooping.