What can cause 'Network is unreachable' when the link is up, the route is there, and the netmask is correct?
What could cause this?
$ sudo ip route show
192.168.100.0/24 dev usb0 scope link
$ sudo ip addr show usb0
85: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UNKNOWN group default qlen 1000
link/ether 16:3d:0a:3f:3b:e6 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.86/24 brd 192.168.100.255 scope global usb0
valid_lft forever preferred_lft forever
inet6 fe80::143d:aff:fe3f:3be6/64 scope link
valid_lft forever preferred_lft forever
$ sudo ip route get 192.168.100.2
RTNETLINK answers: Network is unreachable
This is on my host device, running Android atop a Linux 4.4 kernel. I have plugged in my usb gadget (a Mendel linux device). The usb0 interface appeared; I raised it and ran udhcpc on it. The gadget assigned me 192.168.100.86/24 and advertised itself over mDNS as 192.168.100.2, so I know the link is working. I downed all the other host interfaces to be sure they weren't stealing the route. But if I try to ping it or ssh to it, I get 'Network is unreachable'.
When I do the same steps on a different, non-android host, it works fine. I know that android has some funky iptables rules (e.g. for per-uid data usage tracking) but I didn't think that would affect the routing table (and flushing them all doesn't seem to help). There's also SElinux, but I thought that was just a filesystem thing. What other obscure (to me) linux feature could be blocking me here?
Edit to add, as requested:
$ ip rule
0: from all lookup local
999: from all fwmark 0xa/0xffff lookup 2454
10000: from all fwmark 0xc0000/0xd0000 lookup 99
10500: from all iif lo oif ccmni1 uidrange 0-0 lookup 1003
13000: from all fwmark 0x10063/0x1ffff iif lo lookup 97
13000: from all fwmark 0xd006c/0xdffff iif lo lookup 1003
14000: from all fwmark 0xc0000/0xc0000 iif lo oif ccmni1 lookup 1003
15000: from all fwmark 0x0/0x10000 lookup 99
16000: from all fwmark 0x0/0x10000 lookup 98
17000: from all fwmark 0x0/0x10000 lookup 97
32000: from all unreachable
$ ip route show table local
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
broadcast 192.168.100.0 dev usb0 proto kernel scope link src 192.168.100.86
local 192.168.100.86 dev usb0 proto kernel scope host src 192.168.100.86
broadcast 192.168.100.255 dev usb0 proto kernel scope link src 192.168.100.86
$ ip route show table 2454
default dev ccmni1 proto static
$ ip route show table 99
$ ip route show table 1003
$ ip route show table 97
$ ip route show table 99
$ ip route show table 98
$
Solution 1:
Android uses a lot of routing rules and tables, probably one per application.
As can be seen, without such rule added and probably its corresponding fwmark, a packet will hit routing rule 32000: unreachable.
It's a bit frail to do something manually over this mechanism. In particular routing rule 10500 allows (only) root to use outgoing interface ccmni1, but oif isn't about allowing a packet to be selected to this interface, it's about allowing a packet from a socket bound to this interface (using SO_BINDTODEVICE) to be selected (oif isn't a direct equivalent of iif which is used for routed packets, and iif lo is also a special case for non-routed packets).
Many rules receive a firewall mark probably set by equivalent complex iptables rules to select specific routing rules per application (and its specific UID too). I guess there are specific Android APIs to register such rules when an application is installed.
If you want to allow root to use first the main routing table thus avoiding the unreachable fate, among multiple possible choices:
ip rule add pref 998 uidrange 0-0 lookup main
Or if you don't care that any user so any application can use usb0 simply:
ip rule add pref 998 lookup main
This probably won't integrate well with the Android system which might shuffle rules around when applications are installed or started and one can't tell what's left with iptables/nftables (or even along SELinux, tc etc.) about blocking access. Even binding to a socket might be restricted by additional mechanism (see for example: CONFIG_ANDROID_PARANOID_NETWORK).