Network traffic doesn't appear to leave the trunk

After a long debate in chat involving MikeyB, Pauska, and ChrisS, the problem ended up being two-fold:

1. A possible bug in CentOS 6 was not changing the module options for the bonding module as part of service network restart, so it wasn't tracking my changes between LACP mode (4) and roundrobin (0).
2. Round-Robin mode doesn't like to work with ProCurve switches.

Once I forced the bonded interface to LACP/802.1qa mode through this command:

ifconfig bond0 down
echo "4" > /sys/class/net/bond0/bonding/mode
ifconfig bond0 up

Both the server and the switch were talking. At that point, starting with only one interface enabled on the switch, traffic started working normally. Enabling a second, third, and finally, the fourth interfaces all kept traffic working.

Ultimately, LACP-mode is what made things work. The clue was that round-robin mode worked when there was only one enabled switch-port in the Trunk. The server survives a reboot and comes up in the correct mode. However, a service network restart does not cause the MODE="4" part of the ifcfg-bond0 file in /etc/sysconfig/network-scripts/ to be take effect. If that mode changes, it'll remain what was set at boot (or more likely, module-load time of the bonding module).