UDP traffic not forwarded from Docker containers -> Docker host
Solution 1:
I figured it out.
We had a Trend Micro (anti-virus) agent running in the SOE which I didn't know about.
Fixing it was as simple as:
# systemctl stop ds_agent.service
# pkill ds_agent
Not exactly sure at this point why it is blocking UDP from containers or how to stop it.
Solution 2:
It seems you have a modprobe install directive that cannot work. Possibly it's a result of incomplete update to RHEL 7.2 or some manual fixes.
Try grep -r bridge /etc/modprobe.d /lib/modprobe.d for starters, or otherwise dig around /etc/modprobe.d or /lib/modprobe.d and try to find where does it define the install rule that calls sysctl -q -w net.bridge.bridge-nf-call-arptables=0 net.bridge.bridge-nf-call-iptables=0 net.bridge.bridge-nf-call-ip6tables=0
This sysctl is clearly in wrong place. It is either superfluous or should appear after br_netfilter, not before. Why? Recently the /proc/sys/net/bridge handling has been moved from the bridge module to the br_netfilter module. This happens with some version of kernel*.rpm, while the contents of modprobe.d directories are distributed with other individual packages. I've verified on my RHEL 7.2:
# modprobe bridge
# sysctl -q -w net.bridge.bridge-nf-call-iptables=0
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
# modprobe br_netfilter
# sysctl -q -w net.bridge.bridge-nf-call-iptables=0 # ok now
I don't see these "broken" rules on my vanilla RHEL 7.1 and their origin is mysterious to me. I've tried:
# modprobe -n -vvv bridge
modprobe: INFO: custom logging function 0x40a130 registered
insmod /lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/net/llc/llc.ko
insmod /lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/net/802/stp.ko
insmod /lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/net/bridge/bridge.ko
modprobe: INFO: context 0xf1c270 released
# echo "install bridge echo example_of_a_modprobe_rule" > /etc/modprobe.d/zzz5.conf
# modprobe -n -vvv bridge
modprobe: INFO: custom logging function 0x40a130 registered
insmod /lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/net/llc/llc.ko
insmod /lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/net/802/stp.ko
install echo example_of_a_modprobe_rule
modprobe: INFO: context 0xeaa270 released
# rm /etc/modprobe.d/zzz5.conf
Update: Looks like xenserver uses a similar modprobe hack. It's a nasty bug to globally change kernel module behavior for everyone whether you actually run xenserver or not; and the bug has fired back at us.
Update 2: Now, you've found that /etc/modprobe.d/dist.conf causes this problem and not docker. Whether you have docker or not, modprobe bridge will always return 1 and print error. Normally dist.conf is a part of module-init-tools package on RHEL6. This file is not supposed to be used on RHEL7. It's not on any of my RHEL7 systems and they run just fine. In RHEL7 the package is kmod and it doesn't contain dist.conf. I would:
rpm -qf /etc/modprobe.d/dist.conf # what package owns this file?
If dist.conf is not owned by package, backup it and delete any lines that don't give you any obvious benefit (maybe even delete the file altogether).
If dist.conf is owned by a package, consider removing/updating that package, since it became clearly buggy in terms of RHEL 7.2 compatibility.