Are there rules for pronunciation of words ending in "-ton"?

First of all, as Peter Shor says in the comments:

You should not consider /t.n̩/ /tn̩/ and /tən/ three different pronunciations. These are all essentially the same, and I am really surprised that any dictionary distinguishes between the first and the second.

These differences may sometimes be used to represent a difference in pronunciation, but more often they are simply transcriptional variants.

Moving on to the words listed: it mainly depends on the history of each word. This is not really very helpful to know, as it suggests that you have to look up the pronunciation of each one individually. That is in fact the safest option. However, sometimes there are word parts that you can identify that have a shared pronunciation between many words.

One of these parts appears to be the -on ending used mainly in words derived from Greek: although it has no single source historically, these words tend to be pronounced similarly with the sound /⁠ɒn/. This ending is discussed more in the answer to this question: Why do photons and protons exhibit such anomalous behavior? It's not clear, but I'll speculate that one reason why these words are pronounced with an unreduced vowel is because they are more "learned" terms, so people base their pronunciation more on the spelling. Another relevant factor is that these are all relatively recently coined words; Marcus_33's answer to that question says:

Proton began appearing in science literature in the 1920's. Every other subatomic particle name has been coined since then - these words are all less than 100 years old. They all originated after the invention of audio recordings, and none are used commonly enough to be changed by regional dialects.

The words cotton and mutton come from French coton and mouton respectively. But they entered English a long time ago (during the Middle English period), as indicated by their altered spelling compared to the French, and the placement of stress (on the first syllable). Since the second syllables are unstressed, the vowel became "reduced" here to a schwa sound /ə/. The schwa, when followed by the sound /m/, /n/, /l/, or /r/, may also be transcribed with the subsequent consonant as a syllabic consonant. This is what /n̩/ means.

For the word wanton, the parts appear to come from Old English, and the word has existed at least since Middle English, so it also has undergone many changes in pronunciation over time, among them reduction of the second vowel.

So the advice I would give: if you can tell that the word came from Greek, or if it is the name of a physical particle, it is probably pronounced with /ɒn/. Otherwise, it is probably pronounced with /n̩/~/ən/ (What I mean with the ~ is that you can use either pronunciation interchangeably). And if you don't know for sure, look it up in a good pronunciation dictionary! (Because there are also words that don't follow the rule I gave, like crouton (from French), which the OED lists as /ˈkruːtɒ̃/, but for which I've always used the pronunciation /ˈkruːtɒn/).