What does tcp_orphan_retries set to 0 mean?

Does setting tcp_orphan_retries to 0 mean there is no limit to retries, or does it mean that it won't retry at all?

It doesn't mean "try forever", it means "don't try at all." This is the server trying to politely tell the client that the server is getting ready to close his socket, and if it would please do an orderly disconnect, or send some more data, that would be wonderful. It will try X times to get the client to respond, and after X, it reclaims the socket on the system side.

Setting that number to 0 would suggest to me that that server is heavily utilized, with a zero tolerance policy for orphans. It may also have been a response to a DDOS: lot of DDOS' work by opening a socket connection and then hanging on to it, doing nothing.

Setting tcp_orphan_retries to 0 is a special case, see tcp_timer.c

 98 /* Calculate maximal number or retries on an orphaned socket. */
 99 static int tcp_orphan_retries(struct sock *sk, int alive)
 100 {
 101         int retries = sysctl_tcp_orphan_retries; /* May be zero. */
 102 
 103         /* We know from an ICMP that something is wrong. */
 104         if (sk->sk_err_soft && !alive)
 105                 retries = 0;
 106 
 107         /* However, if socket sent something recently, select some safe
 108          * number of retries. 8 corresponds to >100 seconds with minimal
 109          * RTO of 200msec. */
 110         if (retries == 0 && alive)
 111                 retries = 8;
 112         return retries;
 113 }