Is there a rule for how to pronounce words such as "dance", "prance", "castle"?

Sorry to be pedantic, but it's not grammar. In linguistics, grammar is the set of structural rules that governs the composition of clauses, phrases, and words in any given natural language [Wikipedia]. The way words are pronounced is phonetics: the branch of linguistics that comprises the study of the sounds of human speech. [Wikipedia].

Pronunciation is one aspect of dialect. It's true that Americans and Southern Englishmen pronounce dance differently, although both can be represented /dɑːns/ because the /ɑː/ symbol between slashes is a broad transcriptional device. A more accurate transcription of the American dance might be [däns] or [dɐns]. Northern English makes the a in dance much shorter than Southern English and it's more like /dæns/. In the English West Country it's different again: [dɐᵊns].

As for rules, well, no: it's custom and practice. The American pronunciation of mass is different to the English, but Northern and Southern English are generally almost identical to each other, with a short vowel (some Southerners pronounce it /mɑːs/ but it sounds affected these days). Because pronunciation of vowels in particular is not matched to spelling — think of bough, cough, dough, tough as obvious examples — it's difficult to come up with any rules for how to differentiate between [ä] [ɑː] [ɐ] [æ] and [ɐᵊ] in dance. And there are some words where rules simply don't apply: one text-to-speech program I used insisted on pronouncing without as /woʊ'ʒiːtɑ/! I still don't know if there is any dialect of English which actually does that.

The only way of learning pronunciation truly accurately is immersion and practice.