When is the subnet mask used?

Neither IP or TCP include the subnet mask in their headers.

The subnet mask is used when receiving traffic - if traffic appears on the wire that is not within a subnet that the interface is assigned, it will drop it. If the interface is placed in promiscuous mode, then all traffic it receives is sent up the TCP/IP stack.

The OS uses the subnet mask to know if it should direct traffic directly to the destination IP (if it's within the subnet) or toward a (default) gateway (if it's not).

As NICs on the hardware level traditionally send and receive Ethernet frames, and not IP packets, I don't think this is necessarily a hardware function of the NIC - though modern NICs have "TCP Offload Engines" (TOEs) that perform much of the processing required by TCP on the NIC itself - those probably filter by subnet in hardware too.

The netmask, associated with a network interface, is used in the following situations:

• Normally (but not necessarily) the routing system automatically creates a route to the subnet of the interface
• IPv4 only: the host determines whether a packet received by an interface is a broadcast to own subnet (and should be processed as such).

These options are detailed in an article "Implications of Netmask Mismatch".

The netmask itself does not directly affect whether the host should send the packet directly to the destination address or to a gateway; it is the routing system which is responsible for this decision (but, as said above, netmasks are normally used for constructing the routing tables).