Can't ping multihomed Linux machine on non-default interface

The problem is that since the default route is through eth2, the ping responses get sent through eth2 even though the requests were received on eth3. (If you tcpdump eth2 you should see the responses being sent.) There's then probably some device which is dropping the packets because they have an invalid source IP for the network they're on. You need some source policy routing in order to get the responses to be sent out the interface they were received on.

1. Create a new routing table (only needs to be done once):

   echo 13 eth3 >> /etc/iproute2/rt_tables
   

2. Add a default route to this new table going out eth3:

   ip route add default via 10.20.0.1 table eth3
   

3. Add a policy rule to use this new table for packets with source address of eth3's IP:

   ip rule add from 10.20.0.2 lookup eth3
   

From a separate network (192.168.3.5/24) I can reach this machine on the eth2 interface (the one with the default gateway), but not the eth3 interface. I can ping the eth3 interface from a router on the same network (10.20.0.1) without a problem.

It sounds like you are missing a route for 192.168.3.5/24 from the 10.30.0/24 subnet. You should add a network diagram and traceroutes for each network from each device.