Pronunciation of endings "-tin" and "-tine"

Solution 1:

Interesting question. No, there is no general rule.

I found this

I would pronounce serpentine and Philistine to rhyme with Valentine, but quarantine and libertine to rhyme with seen. I say clan-DES-tin, but some people would rhyme it with Valentine.

Some of these pronunciations must have been influenced by the distinctive pronunciation of Latin that was used by English speakers before the 20th century. https://en.wikipedia.org/wiki/Traditional_English_pronunciation_of_Latin

Solution 2:

I always thought that endings with -tin are pronounced as /tɪn/, like for example in Martin, bulletin or carotin.

Words that end with -tin are usually pronounced as /tɪn/.

Examples: Martin, bulletin, chitin, Latin etc.

Words that end with -tine are usually pronounced with /ʌɪ/ and sometimes with /iː/. The vowel before n in -tine constructions either changes to a diphthong (/ʌɪ/) or a long vowel (/iː/). The pronunciation changes because of the addition of the final e (also called magic e or silent e). The final e changes the vowel to a diphthong or a long vowel.

Examples:

• Hat (/hæt/) + e -> hate (/heɪt/)
• Rat (/ræt/) + e -> rate (/reɪt/)
• Mate (/mæt/) + e -> mate (/meɪt/)

In these examples, the short vowel /æ/ changes to a diphthong /eɪ/ because of the addition of magic e.

• Hop (/hɒp/) + e -> hope (/həʊp/) ——— e changes /ɒ/ to /əʊ/

• Pip (/pɪp/) + e -> pipe (/pʌɪp/) ——— e changes /ɪ/ to /ʌɪ/

Exceptions:

• Love -> /lʌv/
• Have -> /hæv/
• Above -> /əˈbʌv/
• And words that end with the suffix -ive are not affected by silent/magic e.
• Examples: active, representative, passive etc.

But what about words like quarantine, serpentine, clandestine, libertine or philistine? Also, how would you pronounce St. Augustine or Constantine, rather as /tɪn/ or /taɪn/?

Sadly, they don't follow any pattern.

• Serpentine, Valentine and philistine are pronounced with /ʌɪ/.
• Constantine is pronounced with both /ʌɪ/ and /i:/.
• Quarantine, Augustine, Christine, astatine are pronounced with /iː/.
• Libertine can be pronounced with /tiːn/, /tɪn/ or /tʌɪn/.
• Clandestine ends with e, yet it's pronounced with /tɪn/ sound.

Do you know if there is a general rule for those endings or is it optional like in the pronunciation of either?

There are no 'rules' in English, only guidance. English pronunciations don't follow spellings. You'll just have to learn the pronunciations as you meet them.

To add another layer, American pronunciation of philistine and serpentine favors /iː/ , and the "short" i is also, er, routine in brigantine, argentine, levantine, and other words where RP favors /ʌɪ/ — Choster