Spelling with 'ought' or 'aught'
Solution 1:
There's no useful rule.
In some cases, it seems that they corresponded to different pronunciations in older forms of English; but this just adds extra information for you to memorize. And as an added complication, it seems that there were irregular changes between the two sounds before they fell completely together in pronunciation. Some words still show variation in spelling, such as nought/naught.
The Oxford English dictionary has the following useful explanation of nought and naught:
The o forms and a forms remained distinct in Middle English because the sequences -augh- and -ough- were phonologically different; however, in some dialects in late Middle English -ough- came to sound like -augh-, and this pronunciation was gradually introduced into standard English during the 17th cent.
That said, here is how we can explain the spelling of some of these words based on etymology. The past participles thought, brought, bought and fought come from Old English geþōht, gebrōht, geboht, and fohten, while the past participle taught comes from Old English getǣht. The past participle caught does not come from Old English (the verb catch is a loanword from French) but according to Wiktionary, this form may have developed by analogy with the Middle English descendant of the Old English past participle gelæht. In Old English, the past tense and past participle forms of these verbs had different vowels, but these differences seem to have been leveled out in favor of the vowels in the past participles along the way to Modern English.
As a result of this, the following "rule" is valid for remembering the modern English spelling of past participles that end in -aught/-ought: the spelling -aught contains the letter "a," and is used in the past participles of verbs that contain the letter "a" in the present tense (teach and catch) while -ought is used in the past participles of verbs that do not contain "a" in the present tense (think, buy, seek, fight, bring).
The Old English form does not always correspond to the modern spelling like this; for example, daughter comes from Old English dohtor. The Oxford English dictionary says that doughter was formerly the usual spelling, and that the spelling daughter probably reflects the southern vowel change of -ough- to -augh-.