Why is the 't' in 'nextdoor neighbour' usually silent? Where's the 't' in 'postman' gone? And why do people say 'guess book' for 'guest book'?

Neat observation. Here is a rule that works for American pronunciation only; I don't know how this works in other dialects:

(Usual in American:) Syllable-final /t/ is replaced by glottal stop before a consonant or a pause.

(Your observation:) This glottal stop is then further reduced to zero when preceded by a voiceless consonant.

Note that in all your counterexamples, the proceeding sound to the glottal stop is either a vowel or a voiced consonant.

My first explanation:
It's called elision or cluster simplification and is common in many languages other than English. In all of the examples two consecutive stop gestures (where the oral airway is completely blocked) are simplified to one, making pronunciation easier. Elision tends to be regressive (i.e., the gesture that is omitted is the first one).

My second explanation (which contradicts the first, but conveniently explains the counterexamples):
The gestures making up /t/ can be reduced when anywhere outside of foot-initial position (see Harris' paper "Release the captive coda" for a similar but better-thought out statement). The form of the reduction will depend on the phonetic context. The first explanation is confused and totally wrong because the effect doesn't occur across foot boundaries, so there is no regressive assimilation.

In word-final position, AE stops can be unreleased. /t/ and /d/ are the only stops that can be second in a stop cluster, so they will often go unnoticed when part of a word-final stop cluster (though your tongue tip may still go up). When /t/ forms a cluster with fricatives /s/, /S/, or /f/, the /t/ is often unreleased, or omitted altogether. I'd wager there's an aerodynamic explanation for this: intraoral air pressure is bled off in producing the fricative, so there's not as much left for the plosive burst. Other things being equal, /t/ (when fully pronounced) should be weaker in a word-final cluster with a fricative than when by itself. When forming a cluster with a sonorant (nasal or liquid), /t/ will often be realized as glottalization and go unreleased. (say melt and can't to yourself and see). Of course, when foot-internal in a disyllabic word like water, /t/ is tapped and no longer a plosive.

This heavily depends on the accent of the individual. In looking at these, I personally pronounce a lot of these, and I'm not a particularly careful speaker (but I'm not a particularly careless one either) and I speak Australian English (North Queensland, but not as lazy as many).

A lot of the losses seem to be due to the lack of aspirated consonants in the English language. Non aspirated consonants can be very hard to pick up to the casual listener (and even those looking for them). Thus in spoken language, these consonants often disappear completely.

Not all of your examples are this situation, but I think it deals with most of the options, and when combined with the aspects that others have mentioned (elision and cluster simplification), I think will explain the majority of these issues.