Comma usage and maintain succinctness

Your punctuation of the sentence works well. In the University of Sussex Guide to Punctuation, for example, the type of comma you used is called the bracketing comma. That type of comma setting off an introductory adverbial phrase is usual. In your example, however, the comma could be omitted without changing the meaning of the sentence, and so the use is a stylistic choice.

Your punctuation (with a comma after born) is faultless. Your wording "The day I was born" is prefatory to the main subject and verb of the sentence ("Granny died"). The comma marks the end of the introductory phrase, and it is entirely appropriate. Neither a semicolon nor an em-dash would work well here—the semicolon because it would negate the subordinate relationship of the introductory phrase to the main part of the sentence, and the em-dash because it would break the continuity between the intro phrase and the main phrase.