How to reset a tcp connection immediately on both ends on a certain condition using linux netfilter/iptables?

You can use xt_RESET, e.g., -j RESET

You can configure your application to have a timeout.

Another solution is to configure TCP KeepAlive and have more frequent checks, like each 10 min.

Some applications implement a KeepAlive at the application level. E.g. SSH, apache.

When the keep alive is send, and connection is closed at remote end you will receive a RST, from remote end.

Statefull firewalls will forget the connection after a period of inactivity. This means that you can get discarded packets and half open connections when you do not have traffic for a while (30min or 1h).

I think is better to ask about your particular problem you are trying to solve, than about the particular solution you are trying to implement.