Blocking a network device from communicating with another device on the LAN
Solution 1:
If two hosts are on the same subnet, the traffic has no reason to go through the router. Your rules are never being applied. The two devices are connected to a switch (or some other Layer 2 networking hardware). Host A says "I want this traffic to go to Host B's IP" and your switch says "Ok, done."
UPDATE: If VLANs are an option, put each host in a separate VLAN. This way, you can enforce routing rules between them and achieve the logical separation you want.